From Notebook To Broker: Building A Reproducible Quant Research-To-Execution Pipeline With Arrow, Polars, DuckDB And FIX

Every quant programme we have been brought in to fix has had the same shape of failure, and it is never the one people expect. It is not that the models were bad. It is that six months after a strategy went live, nobody could reproduce the backtest that justified it. The data had been updated in place. The feature code had moved. A vendor had revised a series. The live system had a fill-handling rule the backtest did not, added in a hotfix nobody wrote down. The strategy was underperforming and the team could not tell whether the edge had decayed, the implementation had drifted, or the backtest had been optimistic all along - which are three completely different problems with three completely different responses.

The good news is that the tooling to fix this became genuinely excellent in 2026. Apache Arrow has spent a decade becoming the invisible standard underneath the analytical ecosystem, and it is now the thing that makes the stack compose: Polars uses it in memory, delivering up to 15x pandas performance at about a third of the footprint; DuckDB reads Arrow tables without deserialising or copying, so moving between a dataframe and SQL costs nothing; ArcticDB handles versioned tick storage at institutional scale; and NautilusTrader runs the same strategy code in a deterministic backtest and in live deployment off a Rust core with a Python strategy API. The tools are no longer the constraint. The workflow architecture around them is, and that is what this playbook covers.

Layer One: Content-Addressed, Immutable Data

The foundation is that datasets are immutable and addressed by content hash rather than by path. This sounds like ceremony until the first time a vendor silently revises three months of history and your February backtest stops matching itself. Write Parquet, partition sensibly, hash the bytes, and never overwrite - a correction is a new version with its own hash and its own knowledge-time, and the old one stays exactly where it was.

import hashlib, json
from pathlib import Path
import polars as pl

def publish(df: pl.DataFrame, name: str, knowledge_date: str, root: Path) -> str:
    """Write an immutable, content-addressed dataset version.
    Returns the content hash, which is what every downstream step records."""
    tmp = root / "_staging" / f"{name}.parquet"
    tmp.parent.mkdir(parents=True, exist_ok=True)

    # Deterministic write: fixed compression, fixed row-group size, sorted
    # columns. Without this the same logical data hashes differently on
    # different machines and the whole scheme is decorative.
    df.select(sorted(df.columns)).write_parquet(
        tmp, compression="zstd", compression_level=3, row_group_size=128_000,
    )

    digest = hashlib.sha256(tmp.read_bytes()).hexdigest()[:16]
    final = root / name / f"kd={knowledge_date}" / f"{digest}.parquet"
    final.parent.mkdir(parents=True, exist_ok=True)
    if final.exists():
        tmp.unlink()               # already published, byte-identical
        return digest
    tmp.rename(final)

    (final.with_suffix(".meta.json")).write_text(json.dumps({
        "name": name,
        "content_hash": digest,
        "knowledge_date": knowledge_date,   # when this became available to us
        "rows": df.height,
        "columns": sorted(df.columns),
        "schema": {c: str(t) for c, t in zip(df.columns, df.dtypes)},
    }, indent=2, sort_keys=True))
    return digest

Layer Two: Point-In-Time Joins, Or Everything Above Is Theatre

Immutable data prevents the past from changing underneath you. It does not prevent you from joining future information onto a historical row, which is the single most common way backtests are made to look good by accident. Fundamentals carry a period and a publication date. Analyst revisions land days after the period they concern. Index membership changes. Corporate actions get restated. A naive join on ticker and date will cheerfully attach a figure published in May to a decision made in March.

The correct primitive is an as-of join on knowledge time, and both Polars and DuckDB support it directly. The discipline is to make it the only join anyone is allowed to write against time-varying data:

import polars as pl

def pit_join(
    signals: pl.DataFrame,      # one row per (instrument, decision_time)
    facts: pl.DataFrame,        # one row per (instrument, knowledge_time, ...)
    *, lag: str = "0s",
) -> pl.DataFrame:
    """Attach only facts that were KNOWN at decision_time.

    The optional lag models the operational reality that a figure published
    at 07:00 is not in your feature store at 07:00:00.001. Set it to your
    measured ingestion latency, not to zero, unless you enjoy backtests that
    outperform their live counterparts for reasons nobody can explain.
    """
    return (
        signals.sort("decision_time")
        .join_asof(
            facts.sort("knowledge_time").with_columns(
                (pl.col("knowledge_time") + pl.duration(seconds=pl.lit(lag))).alias("available_time")
            ),
            left_on="decision_time",
            right_on="available_time",
            by="instrument",
            strategy="backward",     # only values already available
        )
    )

# The equivalent in DuckDB, for teams whose feature logic lives in SQL.
# DuckDB reads the Polars frames as Arrow with no copy, so mixing is free.
PIT_SQL = """
SELECT s.*, f.* EXCLUDE (instrument, knowledge_time)
FROM signals s
ASOF LEFT JOIN facts f
  ON s.instrument = f.instrument
 AND s.decision_time >= f.knowledge_time + INTERVAL 90 SECONDS
"""

• Ban plain joins on time-varying data in review. This is a lint rule, not a convention - conventions do not survive a deadline.
• Model your real ingestion lag rather than assuming zero. Measure it from your own pipeline telemetry; teams are consistently surprised by their own p99.
• Construct universes point-in-time as well. A backtest run over today's index constituents has survivorship bias baked in before a single feature is computed, and no amount of careful feature engineering repairs it.
• Store the lag in the run manifest. When a backtest and live diverge, ingestion latency is one of the first three suspects and you want the number, not a memory of it.

Layer Three: The Run Manifest

A run manifest is the artefact that makes a result reproducible. It records every hash that can affect an output, is written before the run starts, and is stored with the result. The rule we hold to is that any input not in the manifest is a bug - including the ones that feel too obvious to record, which are exactly the ones that change silently.

from dataclasses import dataclass, asdict
import hashlib, json, subprocess, sys

@dataclass(frozen=True)
class RunManifest:
    run_id: str
    git_commit: str          # code
    git_dirty: bool          # refuse to publish a dirty run to production
    dataset_hashes: dict     # name -> content hash
    config_hash: str         # strategy params, costs, slippage model
    universe_hash: str       # point-in-time membership used
    ingestion_lag: str
    env_lock_hash: str       # lockfile digest - library versions change results
    python_version: str
    random_seed: int
    started_at: str          # supplied by the caller, never Date.now() inside

def build(dataset_hashes, config, universe_hash, ingestion_lag, seed, started_at):
    commit = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()
    dirty = bool(subprocess.check_output(["git", "status", "--porcelain"]).strip())
    cfg = hashlib.sha256(
        json.dumps(config, sort_keys=True).encode()
    ).hexdigest()[:16]
    lock = hashlib.sha256(open("uv.lock", "rb").read()).hexdigest()[:16]

    m = RunManifest(
        run_id=hashlib.sha256(
            (commit + cfg + json.dumps(dataset_hashes, sort_keys=True)).encode()
        ).hexdigest()[:12],
        git_commit=commit, git_dirty=dirty,
        dataset_hashes=dataset_hashes, config_hash=cfg,
        universe_hash=universe_hash, ingestion_lag=ingestion_lag,
        env_lock_hash=lock, python_version=sys.version.split()[0],
        random_seed=seed, started_at=started_at,
    )
    return m

def assert_publishable(m: RunManifest) -> None:
    if m.git_dirty:
        raise RuntimeError("refusing to publish a run from a dirty working tree")

Layer Four: Honest Statistics On Top Of Reproducible Data

Reproducibility makes a result stable; it does not make it true. The other half of the discipline is accounting for how many things you tried. A research pipeline that makes it cheap to test strategies - which is the entire point of the stack above - also makes it trivially easy to generate an impressive Sharpe ratio by selection. If you test two hundred variants and report the best, the headline number is a statement about your search, not about the market.

• Log every trial, including the abandoned ones, against the run manifest. The trial count is an input to your significance calculation, and a pipeline that only records winners has destroyed the denominator.
• Report a deflated Sharpe ratio - adjusted for the number of independent trials and for the non-normality of returns - alongside the raw figure. Bailey and López de Prado's work is the standard reference and the maths is a page.
• Keep a genuinely untouched holdout period that is used once, at the end, with the result published whatever it says. A holdout looked at three times is a training set with a good reputation.
• Include realistic costs and a slippage model in the backtest itself rather than as a haircut applied afterwards. The interaction between a strategy's turnover and its costs is not linear, and a flat haircut systematically flatters high-turnover strategies.

Layer Five: The Parity Harness

This is the piece almost nobody builds, and it is the one that prevents the slow divergence that kills programmes. The premise is simple: if the same strategy code is fed the same market data, the backtest engine and the live engine should produce the same decisions. Any difference is either a bug or an undocumented behaviour, and you want to find out which in CI rather than in production.

Platforms like NautilusTrader are built around exactly this property - a Rust-native core with deterministic backtesting and live deployment sharing one strategy API - and if you are starting fresh, adopting a platform with that design is the highest-leverage decision available. If you have an existing stack, you can still build the harness around it, and it is a week of work:

import pytest, polars as pl
from engine import BacktestEngine, LiveEngine, ReplayFeed

REPLAY_DAYS = ["2026-03-12", "2026-04-28", "2026-06-03"]  # incl. two volatile days

@pytest.mark.parametrize("day", REPLAY_DAYS)
def test_backtest_matches_live_on_replay(day, strategy_config, manifest):
    feed = ReplayFeed(day=day, dataset_hash=manifest.dataset_hashes["l1_ticks"])

    bt = BacktestEngine(strategy_config, clock="simulated").run(feed.clone())
    # The live engine runs against a paper gateway on the SAME replayed feed,
    # with its real order handling, real risk checks and real fill logic.
    lv = LiveEngine(strategy_config, gateway="paper").run(feed.clone())

    decisions = lambda r: pl.DataFrame(r.decisions).select(
        ["ts", "instrument", "side", "qty", "order_type", "limit_price"]
    ).sort(["ts", "instrument"])

    diff = decisions(bt).join(
        decisions(lv), on=["ts", "instrument"], how="outer", suffix="_live",
    ).filter(
        (pl.col("side") != pl.col("side_live"))
        | (pl.col("qty") != pl.col("qty_live"))
        | (pl.col("limit_price") != pl.col("limit_price_live"))
        | pl.col("side").is_null() | pl.col("side_live").is_null()
    )

    # Tolerance is zero for decisions. Fills may legitimately differ - the
    # paper gateway models a real queue - but a DECISION divergence means the
    # two engines disagree about the strategy, which is never acceptable.
    assert diff.height == 0, f"{diff.height} decision divergences on {day}:\n{diff}"

1. 01Pick replay days deliberately: one quiet, one volatile, one with a corporate action or an auction, one with a feed gap. The gap day finds more bugs than the other three combined, because that is where the two engines' assumptions about missing data diverge.
2. 02Assert on decisions, not on fills. Fills legitimately differ when the paper gateway models queue position. A decision difference is a defect by definition.
3. 03Run it in CI on every strategy change and every engine change. The value is entirely in catching drift the day it is introduced, when the diff is one commit wide.
4. 04When it fails, the fix is nearly always to delete a behaviour from one engine rather than to add one to the other. Divergence is a symptom of duplicated logic, and duplicated logic is the disease.

How The Pieces Fit

End to end, the workflow we deploy for quant teams looks like this, and the thing worth noticing is how much of it is plumbing rather than modelling. That ratio is correct, and teams that resist it are the teams who cannot reproduce their own results eighteen months later:

1. 01Ingest to immutable, content-addressed Parquet with an explicit knowledge date. Vendor corrections arrive as new versions, never as overwrites.
2. 02Build features with Polars over Arrow, using as-of joins exclusively for anything time-varying, with a modelled ingestion lag.
3. 03Query and aggregate with DuckDB where SQL is clearer, reading Arrow with no copy - the research layer should feel like one system, not three.
4. 04Write the run manifest before execution; refuse to publish a run from a dirty tree.
5. 05Backtest with realistic costs and slippage, logging every trial including the failures, and report deflated statistics alongside raw ones.
6. 06Promote to live via the parity harness, which must be green. Same strategy code, different engine, identical decisions.
7. 07Execute through FIX or a broker SDK with the same pre-trade risk gate human flow traverses, tagging every order with the run identifier so live performance attributes back to the research that produced it.
8. 08Monitor live-versus-expected continuously, with the manifest available on every position, so 'has the edge decayed or has something changed?' is a query rather than an investigation.

The Bottom Line

The 2026 quant stack is genuinely good - Arrow underneath everything, Polars for speed at a third of the memory, DuckDB for zero-copy SQL, Rust-cored platforms that run the same strategy in replay and in production - and none of it protects you from the failure that actually ends quant programmes: a live system and a backtest that quietly stop being the same thing. The architecture that protects you is unglamorous and mostly plumbing. Make data immutable and content-addressed so the past cannot change. Join point-in-time, with a real ingestion lag, so the future cannot leak backwards. Hash code, config, data and environment into a run manifest so every result has an identity. Count your trials and deflate your statistics so the number you report is about the market rather than about your search. And run a parity harness in CI so backtest and live divergence is a failed build rather than a bad quarter. Build those five things and the fast tools finally pay off, because you can trust what they tell you. That is the workflow automation architecture we build for trading and quant teams in London - and the day it earns its keep is the day someone asks why the strategy is underperforming and you can answer in an afternoon.

References & Further Reading

• Apache Arrow - project documentation and columnar format specification. arrow.apache.org/docs
• The State of Apache Arrow in 2026: ten years in, the invisible standard is everywhere. amdatalakehouse.substack.com/p/the-state-of-apache-arrow-in-2026
• Polars - user guide (lazy evaluation, as-of joins). docs.pola.rs
• DuckDB - ASOF joins documentation. duckdb.org/docs/stable/sql/query_syntax/from.html
• Open Source For You - Polars + DuckDB: the new power combo for in-process analytics (March 2026). opensourceforu.com/2026/03/polars-duckdb-the-new-power-combo-for-in-process-analytics
• NautilusTrader - open-source algorithmic trading platform (Rust core, deterministic backtesting, live deployment). nautilustrader.io
• NautilusTrader on Hacker News - community discussion. news.ycombinator.com/item?id=44810552
• Bailey & López de Prado - The Deflated Sharpe Ratio: correcting for selection bias, backtest overfitting and non-normality (Journal of Portfolio Management). davidhbailey.com/dhbpapers/deflated-sharpe.pdf
• ArcticDB - versioned time-series storage for quantitative research. docs.arcticdb.io

Code in this guide is illustrative and supplied without warranty. Review and test it before production use. Nothing here is investment advice. Important information