Inside BitBank · Engineering an Autonomous AI Research Lab · 8 of 10

Production ML: Serving, Monitoring, Drift, and Rollback

2026-07-24 · 12 min read

A validated model becomes useful only when production can reproduce its inputs, serve it reliably, explain its version, and retreat safely when reality disagrees.

Research code answers, “Can this idea work under the experiment?” Production code must answer a larger set of questions. Can it start after a reboot? Can it survive a missing upstream? Can it reproduce the feature exactly? Can we identify the artifact behind a forecast? Can we roll back without rebuilding the past?

Promotion Produces a Bundle

A model is not one weight file. The deployable unit includes preprocessing, feature definitions, context bounds, numerical types, decision policy, metadata, training-data manifest, evaluation report, and compatibility version. Separating any of these can silently change behaviour.

Promotion should create an immutable artifact with a checksum. Production selects a version; it does not train in place. This makes rollbacks real and lets an incident be reproduced against the exact bundle that served it.

Use the Right Runtime for Each Layer

BitBank's application layer is written in Go, which is well suited to network services, concurrency, predictable deployment, and operational control. Some forecasting libraries remain strongest in Python. A narrow sidecar boundary lets each runtime do what it does best without turning the whole service into one process.

The boundary must be explicit: request schema, timeout, units, model version, error shape, and health semantics. A sidecar should not accept an ambiguous bag of numbers and return an unexplained array.

Warm Paths, Cold Paths, and Capacity

Inference latency includes more than computation. Model loading, graph compilation, cache population, network setup, and first-use allocation can dominate a cold request. Services should measure cold and warm behaviour separately.

Shared GPUs require admission control. Text, image, speech, and research jobs have different memory and latency profiles. Bounded permits and priority tiers prevent two individually safe jobs from racing for the same memory. Background work should use idle capacity and yield to interactive requests.

Health Is Not Readiness

A process can be alive while its model is absent, data is stale, or an upstream is unreachable. Health endpoints should distinguish liveness from readiness and expose the dependencies that matter. Orchestrators can then restart a dead process without sending traffic to one that is merely booting.

Observe the Decision Chain

Useful observability connects collection time, feature version, model version, forecast, policy output, response latency, and eventual outcome. Aggregate dashboards show trends; structured traces let us investigate one decision.

Sensitive inputs and credentials must not enter logs. Reproducibility comes from identifiers and hashes, not from dumping private state.

Four Kinds of Drift

Data drift means inputs changed distribution. Concept drift means the relationship between inputs and outcomes changed. Performance drift means decision metrics weakened. Operational drift means the serving environment—latency, numerical output, dependency versions, or data freshness—changed.

These can occur independently. A price feature may retain its distribution while its predictive relationship vanishes. A model may remain statistically sound while a delayed collector makes every signal stale. Monitoring needs separate indicators and response playbooks.

Delayed Truth

Forecast outcomes arrive later. A one-day horizon cannot be scored immediately, so production monitoring uses leading and lagging indicators. Leading checks cover freshness, ranges, confidence, turnover, and parity. Lagging checks score forecasts and simulated decisions once labels mature.

Shadow, Canary, Champion

A challenger can run in shadow mode without influencing the product. Its live forecasts reveal latency, feature parity, and regime behaviour. A canary exposes a bounded slice when the system supports it. Only then does a reviewed artifact become champion.

The previous champion remains available. Rollback should be a version change with known consequences, not an emergency coding session.

Failure Should Be Boring

Timeouts, retries, circuit breakers, bounded queues, and fallback responses make failure predictable. Retries need care: retrying an overloaded model can deepen overload. Queue limits should reject excess work rather than convert a brief spike into minutes of stale responses.

Continuous Evaluation, Deliberate Release

Benchmarks can rerun daily, drift can be measured continuously, and agents can prepare challenger reports. Production changes should still be deliberate. The release boundary is where evidence, operational safety, and human accountability meet.

Production generates new evidence: failures, latency profiles, drift episodes, and live outcomes. Chapter nine shows how coding agents and institutional memory can turn that evidence into an increasingly capable research organization.