Securely Storing User Broker API Keys

User broker API keys must be encrypted at rest using a per-user Fernet key, itself derived from a master secret stored securely outside the database. This ensures that a database dump alone does not expose usable credentials, making a database leak insufficient for compromise.

What went wrong

We experienced a production incident where a compromised database backup exposed user API keys. While the database itself was not directly breached in real-time, the backup, which contained sensitive user credentials, fell into the wrong hands. The immediate fallout was the potential for unauthorized access to user trading accounts. The cost wasn't just the immediate remediation effort, but the erosion of trust and the significant engineering resources diverted to address the vulnerability. This incident underscored that simply encrypting data at rest, without considering the key management strategy, is a critical oversight.

Why it happens

The fundamental flaw lies in the accessibility of the encryption key. If the key used to encrypt data at rest is stored alongside the encrypted data, or in a location easily accessible from the database, then a compromise of that storage location effectively decrypts everything. This is a common pitfall: treating the database as the sole security boundary for credentials. In our case, the encryption key was accessible from the same environment as the database, meaning a backup of the database also meant a backup of the key needed to decrypt it. This created a single point of failure.

What we changed

Our revised strategy centers on a layered security approach for broker API keys. Per-user broker credentials are now encrypted with Fernet, a symmetric encryption library. The critical element is the encryption key itself: it is derived through PBKDF2 (Password-Based Key Derivation Function 2) from a secret that is held outside the database, typically in environment variables or a dedicated secrets management system. This separation is paramount. Because the encryption key lives in the environment rather than the database, a database dump on its own does not yield usable credentials. The trading path was also re-architected. Keys are registered by the user and resolved at execution time by a per-user broker manager. This means the trading path never needs a global credential for users who supply their own. When no global broker is configured, the executor is constructed with no broker at all and resolves one per user at execution time. This requires that the null-broker path has to be a tested path, not an error path.

Furthermore, we implemented structured logging with a sanitizing processor. This processor strips credential-shaped values from log lines before they are persisted. The most common way secrets escape is a log line, not a direct attacker, and this mitigates that risk. For remote broker gateways, we now require the user's public address rather than localhost. This introduces a second exposure surface: firewall and trusted-IP configuration on the user's own machine, which is a trade-off we accepted for increased security.

How to check your own system

1. Key Storage Location: Verify that your database encryption keys are not stored within the database itself or in a location easily accessible from the database server. They should reside in a separate, more secure environment.
2. Key Derivation: Confirm that your encryption keys are derived using a strong key derivation function like PBKDF2 from a master secret.
3. Execution-Time Resolution: Ensure your trading system resolves user-specific credentials at execution time, rather than relying on a global cache or readily accessible store.
4. Logging Sanitization: Implement a logging processor that actively identifies and strips potential credential formats from log output.
5. Remote Gateway Configuration: If applicable, review the network configuration requirements for remote broker gateways, ensuring they don't inadvertently expose credentials through overly permissive access.

Disclaimer

This article describes engineering decisions in a trading system. It is not investment advice. Trading involves substantial risk of loss and is not suitable for all investors. Past performance does not guarantee future results. Always do your own research and consider your financial situation before trading.