Inside BitBank · Engineering an Autonomous AI Research Lab · 2 of 10

Market Data as Research Infrastructure

2026-07-24 · 12 min read

Most model failures begin below the model: a timestamp with the wrong meaning, a missing candle silently filled, or a research feature that production cannot reproduce.

Models receive most of the attention in machine learning, but market research inherits the truth of every dataset beneath it. If a candle is timestamped by close time in one service and open time in another, the model can appear prescient. If a missing bar is filled using future information, a backtest can gain an edge that never existed.

A Timestamp Is a Contract

Every row needs more than a date. We need to know when an event occurred, when the venue published it, when our collector observed it, and when the strategy was allowed to act. Those moments are not interchangeable.

For a one-hour candle, the high, low, close, and final volume are not known at the opening timestamp. A causal system may calculate a signal after the candle closes and execute at the next available price. Shifting that decision by one row is a small code change with an enormous effect on validity.

Raw, Normalized, and Research Layers

A robust pipeline keeps distinct layers. Raw records preserve what the exchange returned. Normalized records use one schema for symbols, timestamps, numerical types, and intervals. Research datasets add causal features and labels. Separating them makes corrections auditable: we can repair a parser without pretending that the source never changed.

Immutable raw data is particularly valuable when an exchange edits history or an API behaves differently across endpoints. Reproducibility requires knowing not merely which query was run but which bytes were available at the time.

Gaps Are Information

A missing candle may mean no trading, a venue outage, collector downtime, symbol migration, or an upstream rate limit. Automatically interpolating every gap erases that distinction. The pipeline should detect expected intervals, classify gaps where possible, and expose data-quality flags to research.

Forward-filling price can be defensible for a narrow use, but forward-filling volume or external indicators may not be. Any fill policy belongs in the experiment definition, not in an invisible cleanup function.

Corporate Actions for Crypto

Crypto does not have conventional stock splits, yet it has its own identity problems: redenominations, contract migrations, wrapped assets, forks, exchange-specific ticker reuse, delistings, and perpetual contracts whose history differs from spot. A symbol string is not a permanent asset identifier.

Point-in-time universes matter too. Evaluating today's largest assets across old history creates survivorship bias. The failures and delistings that disappeared from the present ranking are part of the opportunity set that existed then.

Layout Determines Research Speed

Data layout is not clerical detail. Columnar formats are excellent when experiments scan a few fields across years. Row-oriented records can be simpler for event replay. Partitioning by venue, interval, symbol, and date changes how much storage must be touched for each run.

Compression can make a pipeline faster, not slower, because modern processors often decompress bytes faster than storage or networking can deliver uncompressed data. Local caches, memory mapping, prefetching, and larger sequential reads keep expensive compute from waiting.

The Feature Contract

Offline and online features must share a contract: name, units, lookback, missing-value behaviour, normalization, numerical precision, and the exact timestamp at which the result becomes available. Ideally the same implementation produces both. When that is impractical, parity tests should compare them over fixed fixtures.

We encountered a concrete version of this problem with live candle limits. If production can retrieve only a bounded history, an offline feature that depends on thousands of earlier bars is not the same feature. Either production must maintain the required state or research must use the live context bound. Tests should prove that changing data outside the declared window cannot alter the current signal.

Labels Can Leak Too

A forward return uses future prices by definition. That is acceptable as a training target, but its horizon changes validation. If a label looks 24 hours ahead, examples near a train-test boundary overlap with the test period. Purge gaps prevent that future from leaking into training.

Cross-sectional labels introduce another trap. Ranking coins at a timestamp is causal only if the full point-in-time universe and every input were actually observable then. Joining a late data feed to an earlier exchange candle can leak even when each table looks individually correct.

Quality Before Quantity

More history is not automatically better. A decade assembled from incompatible venues can be less useful than three years with strong provenance. Data tests should cover uniqueness, monotonic timestamps, expected frequency, numerical ranges, price relationships, stale sequences, extreme jumps, and source freshness.

The final output is a dataset with an evidence trail: source, version, transformations, code revision, and checksum. That lineage allows an experiment to be replayed and a production incident to be traced.

Data Is the First Model

Decisions about sampling, universes, fills, and normalization already encode a view of the market. In that sense, the data pipeline is the first model. Treating it with the same seriousness as a neural network removes an entire class of false discoveries before training begins.

Once the causal foundation is sound, we can compare forecasting approaches on equal terms. Chapter three looks at how simple baselines, boosted trees, and time-series foundation models fit into one evidence-driven model portfolio.