System Status
Real-time health monitoring for all Tradewink services.
Status Command Center
Checking platform modules...
Modules
6
Operational
0
Average Latency
—
Checks
Pass
Service Modules
Probe every 60 secondsCore
Trading Engine
Core orchestrator, market loops, and strategy execution.
Signal Plane
Signals Service
Signal generation endpoints and readiness checks.
AI Layer
Predictions Service
Market forecasting and prediction workflows.
Data Layer
Data Miner
Web scraping, sentiment analysis, strategy backtests, and LLM benchmarks.
Intelligence
Market Intel
Autonomous market analysis, regime classification, and rule distillation.
Presentation
Web App
Customer-facing frontend rendering and routing.
Control Modules
Loading live data
Loading live data
Loading live data
Loading live data
Loading live data
Loading live data
Incident History
Every publicly disclosed incident, written the way we would want a vendor to write it to us: real dates, the real root cause, and what actually shipped to fix it. This history is maintained by hand — we do not yet have automated uptime measurement, so we will not show you an uptime percentage we cannot prove.
Day-trade pipeline outage — no autonomous scans or alerts
Jun 13, 2026 – Jul 9, 2026
26 days
Impact
The autonomous day-trade pipeline produced zero scans, zero trade evaluations, and zero alerts for 26 days. Users relying on autonomous day trading received nothing during that window. The website, dashboards, and published signals continued to operate.
Root cause
Three stacked failures. A Fly.io VM downgrade moved the trading bot from dedicated to shared vCPUs, leaving it CPU-constrained (the downgrade kept its memory allocation; a separate, pre-existing memory leak — fixed independently — added its own pressure). Under that load, concurrent Finviz HTML parsing through lxml hit a libxml2 thread-safety bug and crash-looped the process with SIGABRT. When the process did stay up, single database pool-acquire timeouts aborted entire scan ticks for all users. Worst of all: health checks kept passing the whole time, so no alert fired — the system failed silently for 26 days.
Fixes shipped
- Serialized Finviz lxml parsing so libxml2 thread-unsafety can no longer crash-loop the bot (merged Jul 9, 2026).
- Scan ticks now fail soft: a database pool timeout serves the last-known watchlist from a 15-minute in-memory cache instead of aborting the whole cycle for every user (merged Jul 9, 2026).
- Added a market-hours dead-man alert: if the day-trade funnel writes zero signals for 2 hours while the market is open and broker-connected users exist, an alert fires to the ops channel. It alerts on silence, not just on errors — the exact failure mode of this outage (merged Jul 9, 2026).
- Marked monitoring, not resolved: the fixes above merged on Jul 9, 2026, and this incident will only be marked resolved once the day-trade funnel demonstrably produces signals during market hours on the deployed fixes.
Discord bot stopped silently after host migration
May 4, 2026 – May 9, 2026
5 days
Impact
The Discord bot process was down for roughly 5 days. Users hit "interaction failed" on Discord commands and onboarding. The website kept serving normally, which made the platform look healthy while the bot was dead.
Root cause
A Fly.io host migration sent the bot a clean shutdown signal, and the machine restart policy only retries on failure — a clean exit (code 0) left the process stopped indefinitely. Because the dead process emitted no logs and no errors, log-based monitoring saw nothing wrong.
Fixes shipped
- Added an hourly watchdog that checks the bot machine state on Fly.io and restarts it if it is stopped — covering the clean-exit case the restart policy does not.
Current Monitoring
What watches the system — including what each monitor cannot catch, and which monitors are so new their first production run is still being verified. Both incidents above were silent failures, so this list favors monitors that alert on absence of activity, not just on errors.
Day-trade dead-man alert (newly shipped)
Built after the June 2026 outage specifically to alert on silence, not just on errors. Merged Jul 9, 2026 and rolling out now: once the deploy carrying it is confirmed live, it runs on the hourly agent check-in and fires a Discord alert to the ops channel when the day-trade funnel has written zero signals in the last 2 hours while the market is open and users with connected brokers exist.
Limits: Brand new — its first successful production run has not yet been verified, so it should not be counted on until it has one. Cooldown of one alert per 4 hours. Covers the day-trade funnel only; it does not monitor published SaaS signals or the website.
Hourly bot watchdog
Checks the Discord bot machine state on Fly.io every hour and starts it if it is stopped. Catches clean-exit deaths (for example after host migrations) that the on-failure restart policy does not cover.
Limits: Only recovers a machine in the stopped state. A process that is running but internally jammed (event-loop or DB-pool stall) is not detected by this watchdog.
Live service probes (this page)
Your browser probes each service health endpoint every 60 seconds while this page is open and shows the result above.
Limits: Point-in-time only — results are not recorded anywhere, so this page cannot show historical uptime.
Error tracking
Sentry captures unhandled exceptions across all services, with PII scrubbing enabled.
Limits: Only catches failures that raise errors. Both incidents above were silent failures that produced no exceptions — which is why the dead-man alert exists.
What we do not have yet
- No automated historical uptime measurement — the incident history above is maintained by hand, and the uptime numbers you see on typical status pages cannot be produced honestly here yet.
- No independent third-party uptime probe. Service checks run from your browser and from our own infrastructure.