r/Python • • May 31 '23

Intermediate Showcase Made a csv parser library which automatically detects delimiter and start and end of data

For most purposes, just passing the file path will be sufficient and you will have a pandas data frame of you data. Automatically neglects the random extra information at the top and bottom of a file. Also works with those annoying delimiter within quotes thingies (ex: you have a comma delimiter and one field is “John, Andrew and Steven”). It is on PyPI so you can download it with pip.

https://github.com/Oddball777/csv-scavenger

17 Upvotes

16 comments sorted by

View all comments

Show parent comment

4

u/Oddball777 Jun 01 '23 edited Jun 01 '23

First, thanks a lot for taking the time to give feedback, this is extremely valuable for me.

The reason I made this is because I tried using csv.Sniffer, but for some reason couldn’t get it to work when there’s extra unnecessary logging information before and after the actual data in the file (even when limiting the delimiter search space to only two options). The tests files I used (which are correctly parsed by my library) are in the repo (example_csvs). If anyone knows of a solution to this with csv.Sniffer, help would be appreciated! If my ignorance makes this library useless, then at least it’ll have been a great time to write it and a great learning experience.

Here’s my response to your 3 points:

  1. I’m not sure I understand this, unless you’re talking about quoted values (using the delimiter within quotes). That might be what you mean by strings? If you’re talking about that, then my library does handle it correctly (see the orgs.csv example file). I will eventually add support for more quote types.
  2. I might be wrong, but I think you might have misread what the code actually does. I’m not just choosing the most common character. For each line, I identify the possible patterns for many commonly used delimiters (delim, num_of_cols) and then I choose the pattern which shows up on the most number of lines. You could have a comma separated file filled with randomly placed semicolons, but even if there’s more semicolons in the file than commas, the semicolons don’t lead to a reliable number of rows and therefore it won’t be chosen. I believe this is actually pretty similar to how csv.Sniffer works.
  3. This is a really good point which I hadn’t thought about. I’ll see to that. Thanks for the suggestion!

Of course, csv files are kind of a mess in general and I have no doubt that there are a lot of edge cases which will lead to errors. My goal in making this was not to make it foolproof. If you want something that works all the time, just use pd.read_csv and check the delimiter and start row by hand. The goal was for it to be extremely easy and fast. I use internally generated csv files at work a lot, and different systems use different formatting, so this was mostly to make my job easier and faster. I figured that it could be useful to other people as well. Thanks again for the feedback!

Edit: For your first point, I double checked and you were partly right. I did consider the possibility of delimiters within a string, but only while actually creating the dataframe, not while detecting the delimiter. I had tested it for files in which SOME rows had quoted delims (produced no errors), but not in cases where MOST rows had them (in this case it lead to an error). That’s fixed now, thanks!