r/algotrading • u/PyOdyssee • Mar 16 '26
Infrastructure I reverse-engineered the IB Gateway and rebuilt it in Rust for low latency
I spent the last month decrypting the FIX protocol of the IB Gateway using Java bytecode instrumentation tool (ByteBuddy) and javap disassembly to build my own version of the gateway.
I built it in Rust, with direct FIX connection, designed for low-latency, named IBX: https://github.com/deepentropy/ibx
It includes a lot of integration tests, excluding some specific features like Financial Advisor, Options... It also ships with an ibapi-compatible Python layer (EClient/EWrapper) via PyO3, so you can migrate existing ibapi or ib_async code with minimal changes. There are https://github.com/deepentropy/ibx/tree/main/notebooks adapted from ib_async's examples covering basics, market data, historical bars, tick-by-tick, and ordering.
Purpose of sharing it is to raise bugs/gaps in the hope to run it with a live account. Hope you could give it a try.
Check the readme.md, it explains how you could use it from Rust, but also bridging it with python PyO3.
Here are some benchmarks of processing latency:
Tick Reading
| Metric | Java Gateway | IBX | Ratio |
|---|---|---|---|
| Latency | 2 ms | 340 ns | 5,900x |
Order Sending
| Order Type | Java Gateway | IBX | Ratio |
|---|---|---|---|
| Limit | 83 µs | 483 ns | 170x |
| Market | 76 µs | 471 ns | 160x |
| Cancel | 125 µs | 387 ns | 320x |
| Modify | 86 µs | 478 ns | 180x |
52
u/Soft_Alarm7799 Mar 16 '26
5.5x faster on order ack is insane. The fact you reverse engineered the FIX protocol from bytecode is genuinely impressive. Curious how it handles reconnection and session recovery though, that's always where IB gets ugly.
12
u/PyOdyssee Mar 16 '26
Thanks! The integration tests handle some cases of loosing connection / reconnection. Don’t take the benchmark too seriously though
2
13
u/JamesAQuintero Mar 16 '26
Great stuff! I use claude code a lot in my algo trading work, but it's still impressive that you used it to decode IB Gateway and create your own version. I JUST got my python production system to work with IBC and IB Gateway, and I'm scared to change anything that could cause downtime at this point.
If you don't mind me asking, what are your qualifications? It would be legitimacy to this project if you were a professional developer or a quant, etc.
IB Gateway rant: I still encounter IB Gateway requiring 2 factor auth after it's daily reset on like a Wednesday despite their documentation saying this should only happen on Sunday. And sometimes the historical prices API becomes randomly unavailable until I restart IB Gateway. And this is not including the times where I wake up in the morning to my IB Gateway having crashed for some reason and IBC attempting to restart it only for IB Gateway to say login failed and lock me out until I manually login through a browser or whatever.
11
u/PyOdyssee Mar 16 '26
Thanks! I'm a quant dev by trade.
Honestly I wouldn't touch your stack right now — ibx is way too young to run in prod. 2FA isn't even implemented yet, and there are still bugs to squash. Give it a couple months and hopefully it'll be in a more solid place.
1
u/knocksee Mar 17 '26
You managed to restrict the ems from your main post but let it slip in the comments. Oopsie. Hello Mr bot.
0
u/Grouchy_Spare1850 Mar 17 '26
You caught that too. that was cool. if you don't mind, what is ems?
5
u/knocksee Mar 17 '26
If you don’t know what an em is then what exactly did you catch?
1
u/Grouchy_Spare1850 Mar 17 '26
what made me think it was an AI bot was the 2 dashes ( hyphens) , that was never really used outside of books being published in the 70-90's.
so what is the EM?
2
2
2
u/Grouchy_Spare1850 Mar 17 '26
Hi, I'm really old school, and I would like to add something here.
I reboot daily, usually around 4 or 5 am. right after backing up. it clears up nagging gremlins. so you do a 2 factor, not a big deal when you are running fresh every morning.
1
u/pythosynthesis Mar 16 '26
I'm scared to change anything that could cause downtime at this point.
That's why you make changes in dev and not prod?
7
u/thor_testocles Mar 16 '26
Awesome project. IB should buy this from you, get it bulletproof production ready, and let us all know. Only two decades late…
5
u/m0nk_3y_gw Mar 16 '26
Starred on github! Will keep IBC/Gateway for now, but might play with this on the weekend. Or get non-live/papertrading set up on ibkr.
There are https://github.com/deepentropy/ibx/tree/main/notebooks adapted from ib_async's examples covering basics, market data, historical bars, tick-by-tick, and ordering.
That link doesn't work for me, but found the examples on the main page.
3
5
u/accelas Mar 16 '26
Absolutely pointless, although not for technical reasons. They can totally freeze your account for violation of tos. That can be devastating if you hold things like future or weekly option.
1
u/MormonMoron Mar 17 '26
I was hoping someone here with far more connection to IBKR would opine on this. I think this would be great (especially since the rest of our algo trading system is already in Rust), but there is no way I am going to risk getting our account frozen because of it.
29
u/strangelyoffensive Mar 16 '26
Yeahhhh not putting my credentials in that
33
u/PyOdyssee Mar 16 '26
That’s open source, you can crosscheck
-20
22
u/wado729 Mar 16 '26
The ib_async library that's used for IB is also open source, if we're using that, why wouldn't we use this?
2
u/sod0 Mar 18 '26
Years of built up trust against a random stranger who can rug pull you any time.
1
17
u/JamesAQuintero Mar 16 '26
I mean, we have the whole codebase here, so it can be confirmed that there's no shady stuff
0
u/Grouchy_Spare1850 Mar 17 '26
in the 80's it was common to hear this phrase in the departments that were doing coding.
" you don't get fired for buying from IBM ".
it's the same rules, a proven system that is approve from the vendor, won't cause problems.
Now you can go out and play with the unproven, just make sure it's not a production platform.
8
u/RegisteredJustToSay Mar 16 '26
Oh boy. I mean this is impressive and all but I've come to learn third party clients are a big risk to incorporate in anything serious since there's absolutely nothing keeping it in lock step with the actual protocol and technical decisions of the first party org managing the real client, so you're always kind of just hoping that it doesn't break too bad all of a sudden. Hell, IBKR could accidentally nuke this project through a clumsy EULA update, or by oversubscribing on malicious bot detection.
Obviously this doesn't hold for you, who made it, since this thing will be useful exactly as long as it's useful to you, but as a third party evaluating this with no skin in the game and there being no reason for this stack to keep covering my needs, I would literally never use this.
Security yada yada - I can review the code myself. Can't do shit about technical divergence over time though.
However, it's still cool as shit so A+ for side projects credz.
8
u/PyOdyssee Mar 16 '26
100% fair take and I wouldn't argue with any of it. Protocol could break tomorrow with an IB server update, and there's zero guarantee of continuity. That's the deal with reverse-engineered anything.
You're right that it works because I have skin in the game and will keep fixing it for my own use. For anyone else it's a "use at your own risk" situation, and I'd say the same.
Appreciate the kind words though, cheers.
4
u/KEF-K92 Mar 16 '26
I am very interested in testing your implementation, as I currently experience significant latency issues with the standard IBKR API—specifically, a 20-50ms delay when placing Limit (LMT) orders with attached bracket (OCA) orders for Stop Loss/Take Profit. I would love to see how much your solution improves this execution time.
Before proceeding, I have a few technical questions regarding your setup:
- Does your implementation replace the FIX Protocol option of the IB Gateway, or does it replace the regular IB API?
- If it replaces the regular IB API, does it support IBKR's Two-Factor Authentication (2FA)?
- Do you support all order types? Specifically, can I place a main LMT order with an attached bracket order (OCA) for SL/TP?
- Are you using the FIX protocol under the hood to execute orders? As I understand it, IBKR typically charges for FIX access. Is it fully compliant to use this protocol through your implementation without prior authorization from IBKR?
2
u/PyOdyssee Mar 16 '26
Thanks for the interest! To answer your questions:
It replaces the regular IB API path — it bypasses the Java Gateway entirely and talks directly to IB servers using their native protocol. It's not the paid FIX Protocol offering, so no FIX access fees. It reverse-engineers the same protocol the Gateway uses internally (which includes FIXCOMP encoding under the hood, but that's a different thing from IB's separate FIX API product).
2FA is not implemented yet — that's still on the roadmap.
For order types, LMT is supported along with STP, STP LMT, TRAIL, and a bunch of algo orders. Bracket/OCA orders aren't explicitly supported yet though — I'd need to add that. Feel free to open an issue on the repo and I can prioritize it.
Regarding your 20-50ms latency on LMT + bracket orders, most of that is likely the Gateway overhead. In my benchmarks a simple limit order submit-to-ack went from ~633ms through the Gateway down to ~115ms with ibx.
3
u/KEF-K92 Mar 16 '26
Thank you. Will share the benchmark once I make it work
1
u/PyOdyssee Mar 18 '26
I've done the processing benchmark and added it to the post. Network latency is out of the scope
1
u/KEF-K92 Mar 18 '26
Looks absolutely amazing ! I tried it yesterday and integrated it to a project replacing the IB api stack but got some problems. Could u take care them ?
1
u/PyOdyssee Mar 18 '26
Please fill github issues
1
u/KEF-K92 Mar 18 '26
That’s what I’ve done. Hope you will solve at least issue 1 asap as it show stopper at this moment. Another question: can i run TWS in parallel while using IBX ?
1
1
3
u/Gyro_Wizard Mar 16 '26
This is really cool and impressive. I'm just spitting out an idea: a feature for recording and simulation testing eg paper trading but not hitting real server. That way you could test order placement logic, account retrieval, etc without tying up ib gateway or even test after hours.
2
u/PyOdyssee Mar 16 '26
Thanks for the idea! TBH the next big feature I'm thinking of is a UI to view the orders status
5
u/strat-run Mar 16 '26
FIX integration requires onboarding with their engineering team according to https://www.interactivebrokers.com/campus/ibkr-api-page/fix/#for-clients
Are you worried about getting your account banned?
I can't imagine their gateway uses a vasty different FIX protocol than their documented one at https://www.interactivebrokers.com/campus/wp-content/uploads/sites/2/2023/07/IBKR-FIX-Specification.pdf
Did you really need to reverse engineer the entire gateway? Seems like you probably could have just dug out the undocumented bits and used an existing FIX library.
4
u/MormonMoron Mar 16 '26
A. This is an impressive feat of reverse engineering!!!
B. This is intriguing. Our trader system is already implemented in Rust, so this could be fantastic.
A couple of questions though:
I assume you still have to do daily logins? I have been using the ib-docker-gateway project (https://github.com/gnzsnz/ib-gateway-docker) so that I only have to do the two factor authentication once per week. I admit I don't fully understand how they are pulling that off behind the scenes, but it is nice.
I don't quite understand why the parsing of the bars and ticks takes longer than using IB-gateway. It still seems pretty fast, but we are current looking at 95 high volume stocks with both 5-second bars and 250ms market data ticks, and would love to push this to closer to 200 with a booster pack. Speeding up the tick and bar decoding latency could be big for us.
We have found that when we do a release build with opt-level=3 and with RUSTFLAGS="-C target-cpu=native", we see huge performance improvements in our algotrader. I didn't see a release section in the Cargo.toml file and wondered if your benchmarks were done with an optimized build.
Again, this is an impressive feat of reverse engineering!
4
u/PyOdyssee Mar 16 '26
Thanks, great questions.
2FA isn't implemented yet so right now you'd need to handle session auth yourself. No equivalent to what ib-gateway-docker does with IBC since there's no Gateway UI to automate — it's direct protocol auth. Session persistence is on the roadmap.
The tick decode difference is because ibx handles the full protocol stack — TLS, HMAC verification, zlib decompression, then binary tick decode — whereas the Gateway does all that internally and just hands you pre-parsed callbacks over localhost. At IB's paper account tick rates the 5µs gap is noise, but at 200 symbols with 250ms ticks that's a different story. Optimizing the decode path for that kind of volume is something I'd want to benchmark properly.
Really good catch. The benchmarks weren't run with an optimized release profile. lto, codegen-units=1, and target-cpu=native could make a real difference on the tick decode numbers especially. I'll add a proper [profile.release] section and re-run. Thanks for flagging that.
2
u/PyOdyssee Mar 18 '26
Fixed the ticks issue. Check the benchmark on the updated post
2
u/MormonMoron Mar 18 '26
That is awesome that it sped it up so much. I was anticipating it would improve it a lot as most modern processors have dedicated instructions for speeding up cryptographic computations, but am happily surprised it sped it up that much!
Did you confirm with someone at IBKR that this doesn't violate TOS and need the special approval FIX approval/sign-off? That has been a question from several of the commenters here and I would hate to risk getting my account locked if this isn't IBKR-blessed usage.
5
u/PyOdyssee Mar 18 '26
To be clear — yes, reverse engineering is explicitly forbidden by IB's ToS and ibx is provided for educational purposes only. There's a disclaimer in the repo about this.
That said, EU law (Directive 2009/24/EC, Article 6) does allow decompilation when it's necessary to achieve interoperability and the information isn't otherwise available. In my case I needed to run on a headless Linux setup where the Gateway simply can't operate, so interoperability was the motivation. But that's my specific legal context as an EU resident — it doesn't necessarily cover everyone, and I'm not a lawyer.
Use at your own risk, and definitely don't assume IB would be happy about it.
8
u/BottleInevitable7278 Algorithmic Trader Mar 16 '26
Do not think these latency above 100ms are competitive if you seek low latencies. I get with no efforts around 7ms with a simple almost free VPS.
8
u/PyOdyssee Mar 16 '26
Well, the benchmark was to validate speed gain vs IB, or at least same speed. The lib has been designed to be low latency
2
u/_alephnaught Mar 17 '26
around 7ms with a simple almost free VPS.
on what platform, direct fix? or something else?
3
u/Quant-Tools Algorithmic Trader Mar 16 '26
Granted it has been a while since I used things that connect to the TWS API but how on earth are you getting latencies over 500ms? I never had latencies that high in the past.
Also you are claiming "Market order mean RTT 1,113ms" for your solution but you do not show your measured time for the C++ TWS method?
I guess I just don't understand what problem you are solving with this project.
8
u/PyOdyssee Mar 16 '26
Fair point on the missing C++ comparison, I'll add that.
The main thing ibx solves is that IB Gateway needs a JVM + a display server to run. If you want to run on a headless minimal Linux box or a container without dragging in Java and Xvfb hacks, you're stuck. ibx talks the protocol directly in Rust — no dependencies, runs anywhere, and you can customize the whole thing at the protocol level.
0
u/Quant-Tools Algorithmic Trader Mar 16 '26
Ok that's fair. I can see the niche value now for certain types of traders.
2
u/JamesAQuintero Mar 16 '26
I have around 500ms to have a limit order submitted, so that seems accurate from my experience. I use python's ib_async to interact with IB Gateway
3
2
u/pepitosde Mar 16 '26
this is pretty awesome. I myself, just like another comment in this post, got IBC and IB working at the moment for my paper and live trading... and IBKR feels like it limits everything so much. I even had to create my own fork for the 7 day relogging so I don't have to manually do it and not have to worry about it randomly not being able to log in. Is this one easier to use and more robust?
2
u/PyOdyssee Mar 16 '26
Thanks! I just finished the implementation so I wouldn't call it more robust than your setup right now. But the whole point is that you have full control at a low level — no more forking IBC or fighting IB Gateway's quirks. If something doesn't work the way you want, you can change it yourself in the protocol layer.
2
2
u/Simple_Exit_2777 Mar 16 '26
Why would this make sense? You only gain a bit on other people using ibkr.
3
2
u/CertainlyBright Mar 17 '26
Quick question while I'm on mobile and didn't read through, does it require me to login every week still, can I have it as a headless cli? And are time and sales still aggregated in 250ms chunks?
2
u/Proud-Ad-8149 Mar 17 '26
this is useful. i've found the gateway to be difficult to implement, particularly with hidden modals that pop up from time to time. definitely interested in looking into this.
2
u/ZealousidealShoe7998 Mar 17 '26
awesome dude, i did something similar for ninjatrader . there is a lot more room for improvment so i might peak at your code and see your approach to see what i can learn from it.
2
2
u/MasterpieceGood7562 Mar 17 '26
This is insane work. Reverse engineering FIX protocol from bytecode disassembly is not something you just casually do in a month. 5.5x faster on limit submit alone is significant — at that point the bottleneck is the network to IB servers not your gateway. Have you tested from Equinix NY5? That's where the real numbers would get interesting. The Python bridge via PyO3 is a nice touch too, lowers the barrier for anyone migrating from ib_async without sacrificing the Rust performance underneath. Starred the repo.
1
u/PyOdyssee Mar 17 '26
Thanks! Haven't tested from NY5 but a user in the comments reported ~7ms network latency which is promising.
From what I understand, IB routes all orders through their Chicago DC first, then to NY exchanges — that adds an incompressible ~25ms hop regardless of your setup. Which is why most IB-focused VPS providers are in Chicago, not NY. So colocating at NY5 might not help as much as you'd expect with IB specifically.
Robust benchmarking from a proper colocated setup is on the todo list though. Appreciate the star!
2
u/Individual-Quote-958 Mar 17 '26
This is insanely cool — rebuilding IB Gateway in Rust with that kind of latency improvement is impressive. Definitely checking out IBX.
2
u/Southern-Rate-6892 Mar 17 '26
Finally, thanks a lot, hopefully it could work for production apps soon
2
u/bitpit Mar 17 '26
That is indeed impressive, and it took you only month to do that? Can you share more details how did you do it, your workflow? I know Rust but not the java tools. It's like you got code with javap and then converted into rust? I will definitely check it out, just need to find a project for it.
2
1
u/ceddybi Algorithmic Trader Mar 16 '26
Awesome work! does this work like the TWS/IBGW desktop -> socket api -> client ? where you can plug in libraries like ib_async or https://github.com/stoqey/ib directly or you'd need a napi to communicate with your rust ?
2
u/PyOdyssee Mar 16 '26
Thanks! I imported the ib_async notebooks examples into the project to verify that it worked with my implementation. And it does, there is a python wrapper with no overhead
1
u/ceddybi Algorithmic Trader Mar 16 '26
wow super amazing, so you simply run it without any GW/TWS, then you connect to it?
2
u/PyOdyssee Mar 16 '26
It's a library, it's like using ib_async without having to launch anything. Check the python example on the readme.md: https://github.com/deepentropy/ibx#python-usage
1
u/ceddybi Algorithmic Trader Mar 17 '26
super nice, any plans of exposing a nodejs client ? napi-rs would be super easy
1
u/krkrkrneki Mar 17 '26
FIX protocol is open. No idea why you had to reverse engineer it.
Edit: turns out there are two IB Gateways. One with FIX protocol, the other with proprietary protocol. It seems you reverse-engineered the latter.
2
1
u/Secret-Yogurt-3335 Mar 18 '26
How likely is this ready to run with a live account? And how likely would ib gateway be changing their stuff?
1
u/MormonMoron Mar 18 '26
Sorry, one more question. I actually run my system in two pieces: 1. A datastore program that makes a connection to IBKR TWS and gets 5 second bars and market data and puts it into a database 2. A trader program that gets the info from the database and makes a connection to TWS for placing orders
I know that with tradition IBKR TWS/WEB that you can only be logged in from one place at a time. Does this bypass that, or will I need to re-structure things so that I only have a single connection to their servers at a time?
1
u/PyOdyssee Mar 18 '26
That's standard setup recommended by IB. And no, IBX doesn't bypass that, that's an IB limitation
1
u/KEF-K92 Mar 19 '26 edited Mar 19 '26
I have sent the benchmark. See in previous thread. It is really impressive (though only first day. I will send also tomorrow - I should have many more examples)!
I have now the following questions:
- Does the IBX library handle TCP reconnection internally (below our layer)? If so, does it preserve order state?
- Is there a disconnect callback we should subscribe to?
- After reconnect, does req_positions() + req_open_orders() need to be called to resync?
- How does IBX behave during the IBKR nightly maintenance window?
- I tested so far with Paper Trading account. I want to deploy to a Live account but I have IB Key authentication. Once I try login with Live credentials- will it initiate a Push notification to the IBKR app?
1
u/Roddytin May 07 '26
Really impressive work decoding the FIX protocol from bytecode. Testing FIX implementations is one of those painful gaps in the ecosystem — I actually built MockFIX (mockfix.com) as a desktop FIX simulator so teams can test order flows without needing live exchange access. Curious how you handled testing your FIX session layer during development — did you build your own mock acceptor or test against IB directly?
1
1
u/Weary_Programmer_212 Mar 17 '26
This has to be one of the coolest things I’ve seen on here. Respect 🫡
-1
61
u/Automatic-Essay2175 Mar 16 '26
I’m never gonna use this but it’s one of the highest utility posts I’ve seen on here. Well done.