PyMC Streaming Variational Inference

Mentors. Chris Fonnesbeck (@fonnesbeck, Nashville) and Rob Zinkov (@zaxtax, Berlin).

Problem. PyMC's ADVI and Pathfinder assume the full dataset fits in memory. Financial tick data, sensor streams, and large panel datasets routinely exceed RAM and break this assumption.

Approach. A DataLoader that wraps an arbitrary Python iterable and feeds minibatches into PyMC's existing ELBO scaling machinery through a pm.Data placeholder, driven by a callback-free Trainer, plus streaming-compatible ADVI and Pathfinder loops with online convergence monitoring.

Timeline. Community bonding May 1–24, coding from May 25; midterm evaluation July 6–10, final submission window August 17–24, 2026.

Status. The DataLoader merged into pymc-extras on August 7 (#698). The Trainer, the streaming Pathfinder, and the convergence callback are open in review; the tick-data tutorial is open in pymc-examples (#892). The final report has the full ledger.

The result. PyMC had no supported out-of-core path for VI: minibatching starts from arrays already in memory, and Pathfinder needs full-data gradients. This adds that path as a tested library layer. On the public Criteo 1 TB benchmark the streaming posterior matches an ordinary in-memory fit across all 14 coefficients, while peak memory stays flat at about 0.7 GB from 1 million to 150 million rows; the in-memory baseline reaches 15.7 GB at 150 million and, at the measured slope, would exhaust a 26 GB machine near 238 million rows — a small fraction of the full 4.4 billion. The gain isn't speed; it's inference on data you otherwise couldn't touch.

The design. The abstraction this needed turned out to be one people already know — PyTorch's DataLoader and Trainer — so that is what it is called. loader.total_size carries N for the ELBO rescaling, len(loader) counts batches per epoch exactly as it does in torch, and there is nothing new to learn.

DataLoader — pymc-extras#698 · merged August 7. Out-of-core minibatches from any re-iterable source: Parquet-native, bounded shuffle buffer, total_size resolved from metadata or one counting pass.

Trainer — #710 · ready for review. Drives ADVI over the stream; validated end to end on 30.7M rows of tick data against a closed-form posterior.

Streaming Pathfinder — #722 · in review. Same-batch curvature pairs and Polyak–Ruppert tail averaging; the PR states where it degrades as plainly as where it works.

CheckLossConvergence — #733 · opened August 11. Loss-based stopping on adaptive block horizons; one premature stop in 350 held-out still-improving runs, at a rate forty times below the weakest real signal measured.

Streaming tick-data tutorial — pymc-examples#892 · open, in review. Hierarchical hurdle–Student-t on 300k synthetic rows streamed from Parquet with the merged loader; runs in under a minute; rendered preview.