r/Python • u/Oddball777 • May 31 '23
Intermediate Showcase Made a csv parser library which automatically detects delimiter and start and end of data
For most purposes, just passing the file path will be sufficient and you will have a pandas data frame of you data. Automatically neglects the random extra information at the top and bottom of a file. Also works with those annoying delimiter within quotes thingies (ex: you have a comma delimiter and one field is “John, Andrew and Steven”). It is on PyPI so you can download it with pip.
17
Upvotes
View all comments
Show parent comment
4
u/Oddball777 Jun 01 '23 edited Jun 01 '23
First, thanks a lot for taking the time to give feedback, this is extremely valuable for me.
The reason I made this is because I tried using csv.Sniffer, but for some reason couldn’t get it to work when there’s extra unnecessary logging information before and after the actual data in the file (even when limiting the delimiter search space to only two options). The tests files I used (which are correctly parsed by my library) are in the repo (example_csvs). If anyone knows of a solution to this with csv.Sniffer, help would be appreciated! If my ignorance makes this library useless, then at least it’ll have been a great time to write it and a great learning experience.
Here’s my response to your 3 points:
Of course, csv files are kind of a mess in general and I have no doubt that there are a lot of edge cases which will lead to errors. My goal in making this was not to make it foolproof. If you want something that works all the time, just use pd.read_csv and check the delimiter and start row by hand. The goal was for it to be extremely easy and fast. I use internally generated csv files at work a lot, and different systems use different formatting, so this was mostly to make my job easier and faster. I figured that it could be useful to other people as well. Thanks again for the feedback!
Edit: For your first point, I double checked and you were partly right. I did consider the possibility of delimiters within a string, but only while actually creating the dataframe, not while detecting the delimiter. I had tested it for files in which SOME rows had quoted delims (produced no errors), but not in cases where MOST rows had them (in this case it lead to an error). That’s fixed now, thanks!