Switching higher-order streams to first-order streams

What is a Stream?

Think Iterator, but asynchronous. An Iterator hands you values synchronously with next(); a Stream hands them to you over time with poll_next() and Poll::Ready or Poll::Pending. Poll::Pending means “not yet”; Poll::Ready(Some(t)) means “here you go”; Poll::Ready(None) means “closed — don’t poll again.” Simple, elegant, and a little picky about when you poke it. Ignore Pin and Context for the moment — they’re the bookkeeping staff you don’t want to see.

Combinators: the little factories

Rust’s Iterators win partly because of combinators: map, filter, enumerate — methods that return new Iterators so you can chain transformations like a chef on a production line. Streams get the same treatment. Methods like map and filter take your Stream and produce a Stream (or the same T, if you’re just filtering). The trick is identical in spirit: return a new stream type that implements Stream and run the transforms inside poll_next.

Flattening — the neat trick, the gotcha

Flatten is where things get interesting. You can have a Stream whose items are themselves Streams — a stream of streams. flatten() turns Stream> into Stream by yielding items from the inner streams as if they came from one source. The signature reads a bit dense, but it’s just saying “my items are streams; give me the items inside them.” It’s satisfying in the way a perfectly solved puzzle is. But it’s also where asynchrony bites: which inner stream do you poll? When do you move on? How do you handle Pending from an inner poll without stalling the outer?

Why it matters

Flattening higher-order streams makes composition powerful: you can build complex async flows out of small parts. But the devil’s in the polling details. Timing, ownership and lifecycle of inner streams, and the decision to drive one inner stream to completion or to switch between inners are all practical concerns that change correctness and performance. In short: the API is elegant; the implementation needs careful choreography. Want predictability? Plan your poll strategy — otherwise your beautiful chain can trip over a single Pending.

Sources: mnt.io, Lobsters