Bot Stopped Trading: The Silent Failure

The Bot Was Dead for Five Days and the Dashboard Looked Fine

Your trading bot stopped trading because it exited cleanly with a status code of 0. This is not interpreted as a failure by most host environments, so no alerts were triggered, no restarts were initiated, and the process simply remained stopped. Absence of errors is not evidence of health.

What went wrong

In one incident, our trading bot process exited cleanly at 08:05 UTC and remained in a stopped state for five days. During this period, users interacting with a Discord button that relied on this bot experienced failed interactions because the underlying process was gone. The outage was completely invisible in our log platform; a dead process emits no new logs, so searches for errors within that window returned nothing. The web machine continued to serve HTTP requests, meaning all external health checks and the website itself reported as healthy. A noisy, unrelated database error storm that occurred hours before the exit was a red herring; it was a separate, already-handled bug and not the cause of the bot's stoppage.

Why it happens

Automated trading systems often rely on host environments or orchestration layers to manage process uptime. These systems typically monitor processes for failures and initiate restarts or alerts when they detect them. However, the definition of "failure" is critical. Many systems are configured to restart processes only when they exit with a non-zero status code, which conventionally indicates an error. A process that exits cleanly with a status code of 0 is, by definition, not considered to have failed. This is a deliberate design choice in many operating systems and container orchestrators to distinguish between an intentional shutdown and an unexpected crash. In our case, a shutdown signal was sent during a routine host migration. Because this signal resulted in a clean exit (code 0), the host's restart policy, which was set to "restart on failure," did not trigger. This policy had been intentionally configured to avoid an earlier zombie-retry loop, but it had the unintended side effect of leaving the bot in a stopped state indefinitely after this specific type of shutdown.

What we changed

We implemented an hourly watchdog process. This watchdog's detection signature is a machine in a stopped state with no stop requested and an exit code of 0. When this condition is met, the recovery verb is start (not restart), ensuring the bot is brought back online. This watchdog now automatically initiates the process, preventing the silent, indefinite stoppage we experienced.

How to check your own system

To avoid similar silent failures in your automated trading systems, consider the following checks:

1. Review Host/Orchestrator Restart Policies: Verify that your process manager or orchestrator (e.g., systemd, Kubernetes, Docker Swarm) is configured to handle clean exits (exit code 0) in a way that aligns with your uptime requirements. Does it restart on any stop, or only on explicit failures?
2. Implement External Uptime Monitoring: Beyond internal application logs, use external services to ping critical endpoints or check the health of your trading bot's API. Ensure these checks are granular enough to detect a lack of trading activity, not just network connectivity.
3. Establish a "Heartbeat" Mechanism: Have your trading bot periodically send a "heartbeat" signal to an external monitoring service. This signal should indicate not just that the process is running, but that it is actively performing its intended function (e.g., attempting trades, processing data).
4. Develop a "Stopped but Not Requested" Alert: Configure your monitoring to alert you if a process is found in a stopped state, but no explicit stop command was issued by an operator or automated system. This is the core of our watchdog's detection.
5. Monitor Exit Codes: Ensure your logging and monitoring capture the exit code of your trading bot processes. This data point is crucial for distinguishing between crashes and intentional shutdowns.

Disclaimer

This article describes engineering decisions in a trading system. It is not investment advice. Trading involves substantial risk of loss and is not suitable for all investors. Past performance does not guarantee future results. Always do your own research and consider your financial situation before trading.