ib_insync Crashes: uvloop and nest_asyncio Conflict

ib_insync, uvloop, and 34,897 Leaked File Descriptors

The Interactive Brokers Python wrapper (ib_insync) can crash with the error ValueError: Can't patch loop of type <class 'uvloop.Loop'> when its synchronous convenience methods are invoked. This occurs because these methods internally attempt to use nest_asyncio to patch the event loop, a process that is incompatible with uvloop. In our production environment, this incompatibility led to a cascade of leaked file descriptors, ultimately forcing an emergency restart.

What went wrong

Our automated trading systems at Tradewink rely on high-performance asynchronous I/O. To achieve this, we deploy uvloop in production. During a routine deployment, we encountered intermittent failures in our trading logic. Specifically, calls to synchronous convenience methods within ib_insync, such as the one used for contract qualification, began failing. These failures were not reproducible in our local development environments, which use Python's default asyncio event loop.

The symptoms were subtle at first: increased latency and occasional hangs. As the system continued to run, the problem escalated. We observed a rapid accumulation of open file descriptors. The process eventually hit the operating system's limit of 65,536 file descriptors, triggering an ungraceful termination. Our monitoring revealed that the process had accumulated 34,897 leaked file descriptors before the crash.

Why it happens

The root cause lies in how ib_insync handles its synchronous convenience methods. These methods are designed to provide a simpler, blocking interface for common operations. To achieve this without requiring the user to explicitly manage an asyncio event loop, ib_insync internally calls nest_asyncio.apply(). The purpose of nest_asyncio is to enable the use of asyncio within synchronous code by patching the event loop.

However, nest_asyncio has a critical limitation: it only knows how to patch the standard library's asyncio event loop. When it encounters a different event loop implementation, such as uvloop, it cannot perform the necessary patching. Instead, it raises a ValueError with the message Can't patch loop of type <class 'uvloop.Loop'>.

In our production setup, uvloop is the active event loop. Therefore, every time one of these synchronous ib_insync methods was called, nest_asyncio attempted to patch uvloop, failed, and raised the ValueError. Crucially, the exception was not always handled gracefully by the calling code, and each failed patching attempt resulted in a resource leak.

What we changed

The immediate and most critical fix was to prevent nest_asyncio from attempting to patch uvloop in our production environment. We implemented a guard at every call site that interacts with a synchronous ib_insync method. This guard inspects the currently running event loop. If the loop's module is identified as uvloop, the synchronous call is bypassed, and a fallback mechanism is invoked.

This fallback mechanism involves either raising a more informative error or, in some cases, returning a default value. The goal is to prevent the ValueError and the subsequent file descriptor leak. This approach effectively stops the bleeding and stabilizes the system.

However, we recognized this as a workaround rather than a durable solution. The underlying issue is the reliance on synchronous methods that internally use nest_asyncio in an environment where uvloop is preferred for performance. The truly durable fix is to migrate away from these synchronous convenience methods entirely.

The asynchronous variants of ib_insync methods are proper coroutines. These are designed to work correctly and efficiently with uvloop without any patching. Therefore, the long-term strategy is to refactor our codebase to exclusively use the asynchronous API. This aligns with best practices for high-performance asynchronous applications and eliminates the compatibility conflict.

How to check your own system

If you are running automated trading systems that utilize ib_insync and uvloop, it is prudent to check for potential exposure to this issue. Here's a simple checklist:

1. Identify uvloop Usage: Confirm that your production environment is indeed running uvloop. This is typically done by checking your dependencies or how your asyncio event loop is initialized.
2. Locate Synchronous ib_insync Calls: Audit your codebase for any direct calls to synchronous convenience methods within ib_insync. Examples include methods like ib.qualifyContracts() or any other methods that do not require await.
3. Monitor File Descriptors: Implement monitoring for open file descriptors on your trading servers. A sudden or steady increase in file descriptor usage, particularly for eventfd or similar types, can be an early warning sign.
4. Review Error Logs: Scrutinize your application logs for ValueError: Can't patch loop of type <class 'uvloop.Loop'> or any related exceptions originating from nest_asyncio or ib_insync.

If you identify synchronous ib_insync calls in a uvloop environment, consider implementing the guard mechanism as an interim solution while planning a migration to the asynchronous API. The ultimate goal should be to eliminate the use of these synchronous methods to ensure stability and performance.

import asyncio
import uvloop
import ib_insync

async def check_loop_compatibility():
    # Ensure uvloop is installed and set as the default policy
    uvloop.install()
    loop = asyncio.get_event_loop()

    print(f"Running with loop: {type(loop).__name__}")

    if isinstance(loop, uvloop.Loop):
        print("uvloop detected. Synchronous ib_insync methods may cause issues.")
        # Example of a synchronous call that would fail if not guarded
        # try:
        #     ib = ib_insync.IB()
        #     ib.connect('127.0.0.1', 7497, clientId=1)
        #     contracts = ib.qualifyContracts(ib_insync.Stock('AAPL', 'SMART', 'USD'))
        #     print(f"Qualified contracts: {contracts}")
        #     ib.disconnect()
        # except ValueError as e:
        #     print(f"Caught expected error: {e}")
        # except Exception as e:
        #     print(f"Caught unexpected error: {e}")
    else:
        print("Standard asyncio loop detected. nest_asyncio patching should work.")

if __name__ == "__main__":
    asyncio.run(check_loop_compatibility())

Disclaimer

This article describes engineering decisions in a trading system. It is not investment advice. Trading involves substantial risk of loss and is not suitable for all investors. Past performance does not guarantee future results. Always do your own research and consider your financial situation before trading.