Market Data Pipeline Resilience

To build a reliable market data pipeline from multiple providers, adopt an async-first architecture with native async methods that directly call the primary provider and seamlessly fall back to secondary sources via an executor upon failure. Layer caching and meticulously test data integrity, especially against fabricated bars.

What went wrong

Our automated trading system experienced a critical failure during a period of high market volatility when our primary real-time market data API experienced an outage. The system, designed to ingest tick and aggregate data, froze, leading to missed trading opportunities and, more critically, a cascade of stale data being processed by downstream analytical components. The core issue was not the primary provider's failure itself, but the system's inability to gracefully handle the interruption and provide a consistent, albeit potentially degraded, data stream. The cost was measured in lost alpha and the significant engineering effort required to diagnose and rectify the data flow.

Why it happens

Market data providers, regardless of their service level agreements or perceived reliability, will eventually fail. These failures can stem from a multitude of reasons: infrastructure issues, upstream data source problems, network disruptions, or even planned maintenance that wasn't communicated effectively. Furthermore, the complexity of real-time data ingestion, involving multiple independent streams like websockets for bars/trades and news, and pollers for regulatory filings, creates numerous potential points of failure. Each stream has its own failure mode, and a burst of data on one stream can overwhelm consumers if not properly filtered. Caching, while essential for performance, introduces its own challenges; cache coherency between distributed processes is not an inherent property and requires explicit validation.

What we changed

We re-architected our market data layer to be async-first. This means our primary provider is called directly using native async methods. When this primary connection fails, our system automatically falls back to a secondary source through an executor. This fallback mechanism is not a single point of truth; we utilize a diverse set of secondary and complementary providers. These include a free historical data source, a news and fundamentals API, a macro series API, and a regulatory filings feed. Each of these has its own distinct failure characteristics, which we've mapped and accounted for in our fallback logic.

Our real-time data flow is now managed through three independent streams: a bars and trades websocket, a news websocket, and a filings poller. Crucially, events from these streams pass through filters before being dispatched to consumers. This prevents a sudden burst of activity on one stream from flooding the entire system. This filtering is essential for maintaining stability.

Caching has been implemented in layers. We utilize a shared cache when available, and an in-memory cache when the shared cache is not accessible. This layered approach necessitates rigorous testing to ensure cache coherency between different processes, as it's not a property that can be assumed.

A significant change was how we handle historical data access. Our historical accessor now returns a wrapper object, not a raw dataframe. Initially, treating this wrapper as a dataframe was a recurring source of bugs. We've since documented the analyzer inputs per method to prevent this class of errors. The most critical failure mode we guard against is the synthesis of fabricated bars. A fallback that creates a "fake" candle to fill a gap, rather than reporting the gap, will silently corrupt every downstream indicator and trading signal.

How to check your own system

1. Provider Failure Simulation: Manually disconnect or simulate failures for each of your market data providers (primary and secondary). Observe how your system reacts. Does it fall back gracefully? Are there any unhandled exceptions?
2. Data Integrity Check: After a simulated failure and recovery, compare the data received from your system against a known good source for the period of the outage. Pay close attention to any gaps or anomalies in bars and trades.
3. Cache Coherency Test: If you have distributed caching, implement tests that verify data consistency across different instances of your application after cache updates and invalidations.
4. Stream Flood Test: Inject a high volume of data into one of your real-time streams (e.g., news websocket) and verify that other streams and consumers are not negatively impacted.
5. Wrapper Object Usage: Review your code that consumes historical data. Ensure you are not treating any data access wrapper objects as raw dataframes without explicit conversion and validation.

Disclaimer

This article describes engineering decisions in a trading system. It is not investment advice. Trading involves substantial risk of loss and is not suitable for all investors. Past performance does not guarantee future results. Always do your own research and consider your financial situation before trading.