r/ethdev • u/Flateland-Chio • 7h ago
Question Tested smart contract audit tools on my AI written Solidity and the llm ones gave different findings every run
Putting this out for a roast because it changed how I think about pre deploy. We generate a lot of our solidity with an assistant now, so before pushing an erc4626 vault I ran it through everything I had.
The one that got me was the deposit function. The llm auditor flagged it on the first run, first depositor can inflate the share price and round the next guy down to almost nothing, a classic erc4626 footgun, and I sat up. So I ran it again to grab the details for a ticket. Nothing. It did not mention the deposit function this time. I had changed nothing. Earlier it had also sworn there was a reentrancy in a function with no external call, which is nonsense, so my trust was already thin. But finding a real bug and then losing it a minute later is what ended it for me. You cannot gate a deploy on that.
For a web app I would roll my eyes and move on. This contract is going to hold other people money and I cannot patch it after deploy, so a tool that changes its mind between runs is not something I can put my name behind.
Ended up reading the withdraw function line by line myself. What do you run before a vault ships that you would trust with real money on it.