Dataflow Pipelines: TPL Dataflow and Rx
Objective
Channel<T> and Parallel.ForEachAsync cover most producer/consumer work, but two older libraries still show up in
real codebases: TPL Dataflow (blocks linked into a pipeline, bounded buffers, pull style) and Rx.NET
(IObservable<T>, push style, built for event streams). This concept builds one pipeline in Dataflow, shows where Rx
differs (cold and hot sources, schedulers, the subscription that is never disposed, no backpressure), covers
Task.WhenEach and the .NET 10 change to async LINQ, and ends with one job run through Channel<T>, Dataflow, Rx and
Parallel.ForEachAsync, measuring throughput and memory. Baselines covered elsewhere:
Channels,
Parallel.ForEachAsync,
IAsyncEnumerable. Ran on macOS arm64 (14 cores).
Versions: .NET SDK 10.0.100, TFM net10.0. System.Threading.Tasks.Dataflow 10.0.0.0 from the shared framework (no
package, see below), System.Reactive 7.0.0, System.Linq.Async 7.0.1 (only for the Rx bridges).
Use Cases
- An in-process pipeline whose stages have their own parallelism and buffer size (parse, enrich, batch, write).
- Batching by count before a bulk insert; event streams with time operators (debounce, buffer, combine latest).
- Consuming results as each task finishes, with
Task.WhenEach.
Deep Dive
Dataflow ships in the framework
On net10.0 it is not a package. Referencing System.Threading.Tasks.Dataflow 10.0.15 gave warning NU1510 ("will not
be pruned ... likely unnecessary") and the assembly loaded at run time came from Microsoft.NETCore.App/10.0.0. The docs
say it is included in .NET 6 and later; the package is for .NET Framework and .NET Standard.
A pipeline: producer, parallel transform, consumer
csharpvar buffer = new BufferBlock<int>(new DataflowBlockOptions { BoundedCapacity = 4 });
var transform = new TransformBlock<int, int>(
async i => { await Task.Delay(i % 5 == 0 ? 40 : 5); return i * i; }, // every 5th is slow
new ExecutionDataflowBlockOptions
{ MaxDegreeOfParallelism = 4, BoundedCapacity = 4, EnsureOrdered = ordered });
var sink = new ActionBlock<int>(async v => { arrivals.Add(v); await Task.Delay(5); },
new ExecutionDataflowBlockOptions { BoundedCapacity = 4 });
var link = new DataflowLinkOptions { PropagateCompletion = true };
buffer.LinkTo(transform, link); transform.LinkTo(sink, link);
for (int i = 0; i < 40; i++) await buffer.SendAsync(i);
buffer.Complete(); await sink.Completion;textEnsureOrdered=True: 371 ms, max concurrent transforms 4, produced-but-not-consumed peak 11, arrivals 0,1,2,3,4,5,... EnsureOrdered=False: 250 ms, max concurrent transforms 4, produced-but-not-consumed peak 10, arrivals 1,3,2,4,6,8,...
Three runs agreed within 5 ms. By default output order equals input order, so a slow item holds back the fast ones behind it (371 ms against 250 ms, where item 0 arrived tenth). Turn ordering off only when order does not matter.
BoundedCapacity is the backpressure
BoundedCapacity caps what a block holds (queued, in process, and finished output waiting for the next block). With 4
in each of three blocks the producer was never more than 11 items ahead. Post and SendAsync differ when a block is
full:
textPost into BoundedCapacity=3 with a slow consumer: accepted 3, rejected 7 SendAsync of 10 items: producer was held for 362 ms (waited for room) Default (unbounded) block: Post x1000 all accepted, InputCount right after = 1000
Post returns false and the item is lost unless you check it; SendAsync waits for room. The default is unbounded,
the same memory trap as an unbounded channel.
Completion, faults and cancellation
PropagateCompletion = true forwards completion, fault and cancellation to the next block; without it the downstream
block never finishes.
texta fault chain: AggregateException -> AggregateException -> InvalidOperationException no PropagateCompletion: t2 RanToCompletion; a2 completed within 500 ms? False ActionBlock faulted on item 2: handled 2 of 10, status Faulted try/catch inside delegate: ok 6, failed 4, status RanToCompletion cancel: t5 Canceled, a5 Canceled, transformed 10 of 100, input left 0
An exception escaping a delegate faults the block for good: the queued items (8 of 10) are dropped and only
Completion says so; Post then returns false. Each propagation wraps the exception in another AggregateException,
so use Flatten() or GetBaseException(). To keep going after a bad item, catch inside the delegate. For cancellation
pass the token in the options of every block; items in flight finish and the queue is discarded.
The other blocks
textafter TriggerBatch: [1,2,3,4,5] [6,7,8,9,10] [11,12] (12 posted, batch size 5) BroadcastBlock, 20 messages: fast target got 20, slow target (30 ms per item) got 5: 0,5,10,15,19; late target got: c
BatchBlock<T>(5)emits arrays of 5; the tail waits forTriggerBatch()orComplete(), so add a timer that callsTriggerBatchif a half batch must not wait forever.BroadcastBlock<T>offers each message to every target but keeps only the latest: the slow target saw 5 of 20 and a late subscriber got onlyc. It is a "current value" block, not a fan-out queue.- Non-greedy
BatchBlock,JoinBlock,TransformManyBlock: docs only.
Rx: push, and cold or hot decides who runs the producer
textcold: 2 subscribers, producer ran 2 times Publish().RefCount(): producer ticks 5; A saw [0,1,2,3,4], late B saw [2,3,4] hot Subject: early [1,2,3], late [3]
A cold observable (Observable.Create, Interval, Range) starts the producer per subscriber. A hot one
(Subject<T>) exists on its own and late subscribers miss earlier values. Publish().RefCount() shares a cold source;
ReplaySubject(2) replays the last two (late subscriber got [2,3]).
Rx operators and schedulers
textThrottle(60ms) over a,b,c (15 ms apart), d, e,f: emitted c@90 d@230 f@370 CombineLatest(1,2 | x | 3 | y): 2x 3x 3y main: thread 1; Range subscriber ran on thread 1 (synchronous); Interval tick on thread 5 (pool)
Throttle is a debounce: it emits the last value after a quiet period (c, d, f), not a rate limiter.
CombineLatest waits for both sides, so the 1 never appears. Range runs synchronously on the subscribing thread,
Interval on a pool thread; ObserveOn moves the callbacks and SubscribeOn the subscription, so a UI app puts
ObserveOn with the UI scheduler right before touching a control. Virtual-time tests use TestScheduler from
Microsoft.Reactive.Testing (docs only).
The subscription that is never disposed
Observable.Interval(10 ms).Subscribe(_ => widget.Ticks++), then the references to the subscription and widget are
dropped and GC.Collect() runs three times:
textsubscription dropped without Dispose: after GC the widget is STILL ALIVE (rooted by the timer), ticks 10 then 21 subscription disposed: ticks at 100 ms = 0; after GC the widget is collected 100000 short-lived subscribers, never disposed: retained 59345 KB, 100 x OnNext took 47 ms 100000 short-lived subscribers, disposed: retained 8 KB, 100 x OnNext took 0 ms
The scheduler's timer, not your variable, keeps the callback and what it captures alive. On a long-lived Subject (or
an event wrapped with FromEventPattern) each forgotten subscription also stays in the observer list. Dispose in
Dispose, keep subscriptions in a CompositeDisposable, or end the stream with TakeUntil(stop).
Rx has no backpressure, IAsyncEnumerable pulls
A fast producer of 200,000 items, the same slow consumer, counting items produced but not yet consumed:
textRx Range(200000).ObserveOn(slow consumer): max items waiting = 199455 IAsyncEnumerable pull, same slow consumer: max items waiting = 0 Channel.CreateBounded(100), same slow consumer: max items waiting = 100
ObserveOn decouples the sides with an unbounded queue, so push never slows the producer. await foreach produces only
when asked, and a bounded channel blocks the producer at a chosen limit. Rx fits sources you cannot pause (clicks,
sensors); pull fits a source you control that can outrun you. The bridges (ToAsyncEnumerable(), ToObservable()) are
not in System.Reactive 7.0.0: they did not compile until System.Linq.Async 7.0.1 was referenced.
Task.WhenEach (.NET 9)
await foreach (var t in Task.WhenEach(tasks)) yields each task as it completes:
textWhenEach yielded the 50 ms task at 50 ms (then 100, 200, 300 ms) task faulted: bad one (enumeration kept going) n= 10000: WhenEach 20 ms 4 MB allocated | WhenAny loop 268 ms 404 MB allocated n= 40000: WhenEach 53 ms 20 MB allocated | WhenAny loop 4410 ms 6139 MB allocated
Tasks come back in completion order, and a faulted task is yielded, not thrown: the exception appears on await t and
the loop goes on. The old pattern calls WhenAny and removes the finished task each time, which is quadratic (same
sums in both).
.NET 10 ships async LINQ
System.Linq.AsyncEnumerable.dll is in the .NET 10 shared framework, so Where, Select and ToListAsync on
IAsyncEnumerable<T> need no package (usage in Async Streams). A
project that still references the old package, same code (Where, Select, ToListAsync, SelectAwait):
textSystem.Linq.Async 6.0.1 and 6.0.3: error CS0121 (ambiguous call) on Where and ToListAsync System.Linq.Async 7.0.1: builds; SelectAwait warns CS0618 ("Use Select ... additional CancellationToken argument")
So upgrade to 7.0.0 or later (the .NET 10 breaking-change page says the same) or drop the reference; SelectAwait
callbacks become Select overloads with a CancellationToken.
Choosing, with a measured job
One job, four implementations, each in its own process, 5 runs, medians (range in parentheses). Items are produced fast
and carry a payload (1 KB for CPU, 4 KB for I/O); concurrency is 14 (CPU) or 100 (I/O). The Rx version needs
Observable.FromAsync(() => Task.Run(...)) so work leaves the subscribing thread, and Merge(maxConcurrent) to cap it.
All four gave the same checksum.
CPU job (200,000 items, a few microseconds of hashing each):
| Approach | ms | items/s | allocated MB | peak items in flight |
|---|---|---|---|---|
Channel<T> bounded 1000, 14 workers |
460 (356 to 498) | 434,783 | 200 | 1,015 |
Dataflow ActionBlock, bounded 1000 |
377 (365 to 386) | 530,504 | 200 | 1,001 |
Parallel.ForEachAsync over Enumerable.Range |
558 (452 to 650) | 358,423 | 213 | 14 |
Rx Select + Merge(14) |
1063 (1015 to 1201) | 188,147 | 417 | 78,461 |
I/O job (20,000 items, Task.Delay(10) each, ideal about 2000 ms), as ms, MB allocated, peak items in flight, peak heap
MB: Channel<T> 2329, 84, 1,101, 29; Dataflow 2347, 90, 1,001, 24; Parallel.ForEachAsync 2398, 87, 100, 7.5; Rx 2400, 110,
19,800, 186.
When the work is a delay the four sit within about 3% of each other, so the choice is memory and shape, not speed. For
CPU work Channel and Dataflow were within noise, ForEachAsync was slower, and Rx took 2 to 3 times as long and
allocated twice as much, with tens of thousands of items created before being done (peak heap 302 MB), because Range
emits everything at once and nothing pushes back. Peak heap is sampled and includes garbage not yet collected.
| If you need | Pick |
|---|---|
| A queue between producers and consumers, simplest code | Channel<T> |
| A loop over a collection or stream with a concurrency cap | Parallel.ForEachAsync (least memory above) |
| Several stages, per-stage parallelism, batching, ordered output | Dataflow |
| Time operators over events you cannot pause | Rx |
Book vs today
- Dataflow lives in
dotnet/runtime(src/libraries/System.Threading.Tasks.Dataflow) and ships in the framework. On 2026-11-20 that folder had 9 commits since 2025-11-20 and 21 since 2024-11-20, mostly repository-wide changes (analyzers, polyfills, CI); the one behavior fix was aWriteOnceBlockrace (#123435, February 2026). The package gets a version per servicing release (10.0.9 to 10.0.15 monthly, 11.0.0 on 2026-11-10). Maintained, not growing. Material from the .NET 4.5 era presents it as the way to build pipelines; today channels andParallel.ForEachAsynccover the common cases. - Rx:
dotnet/reactiveis not archived, had commits on the day of this check (17 since 2025-11-20, 47 since 2024-11-20) and released 7.0.0 on 2026-07-17 (RC 2026-06-18; 6.1.0 was 2025-10-03). It has nonet10.0target (usesnet8.0). - Adoption signals, not job postings. NuGet all-time downloads (include transitive restores):
System.Threading.Channels1.66 B, Dataflow 898 M,System.Reactive469 M,System.Linq.Async404 M. GitHub code search, C# files:using System.Reactive.Linq62,080;Channel.CreateBounded22,496;Parallel.ForEachAsync13,872;ActionBlock<3,960;Task.WhenEach415 (Rx is heavy in UI code, not separated here). Job postings: not measured.
Trade-offs
- Dataflow is verbose. Options on every block and nested
AggregateExceptions are more ceremony than a channel with workers; the payoff is per-stage capacity, ordering and batching. Ordering cost 371 ms against 250 ms. - Bound every block and use
SendAsync(7 of 10Postcalls were rejected above). A faulted block drops its queue, so catch per item inside the delegate and watchCompletion. - Rx moves the problem to the consumer. Push without backpressure queued 199,455 items; use it for events, not a
fast finite source you control, and dispose every subscription (59 MB retained otherwise).
Throttledebounces; useSampleor a token bucket to limit a rate. - Rx is a second vocabulary (cold versus hot, schedulers,
Publish), andSystem.Linq.Asyncbelow 7.0 does not compile against async LINQ on .NET 10.
Documentation Links
- Dataflow (Task Parallel Library), Microsoft Learn
- Walkthrough: Implement a producer-consumer dataflow pattern, Microsoft Learn
- DataflowLinkOptions.PropagateCompletion, Microsoft Learn
- Rx.NET repository, GitHub
- Introduction to Rx.NET, introtorx.com
- Task.WhenEach, Microsoft Learn
- Breaking change: System.Linq.AsyncEnumerable in .NET 10, Microsoft Learn
- AsyncEnumerable class, Microsoft Learn