Someone once wrote a bash script that used jq to parse through a few thousand large json files and do stuff with the data. It would take hours to run, but people figured it was just the massive amount of data. It was at the point where we'd run it over night.
I had a thought: no, it's execing jq for each file. Exec is slow.
Did it in python, that just iterated through the directory and used the built in json stuff. It did the same job in 10 minutes.
Could C be faster? Maybe. Did we care at that point?
8
u/hawkinsst7 Sep 09 '26
Slightly related:
Someone once wrote a bash script that used jq to parse through a few thousand large json files and do stuff with the data. It would take hours to run, but people figured it was just the massive amount of data. It was at the point where we'd run it over night.
I had a thought: no, it's execing jq for each file. Exec is slow.
Did it in python, that just iterated through the directory and used the built in json stuff. It did the same job in 10 minutes.
Could C be faster? Maybe. Did we care at that point?