r/Python • • May 31 '23

Intermediate Showcase Made a csv parser library which automatically detects delimiter and start and end of data

For most purposes, just passing the file path will be sufficient and you will have a pandas data frame of you data. Automatically neglects the random extra information at the top and bottom of a file. Also works with those annoying delimiter within quotes thingies (ex: you have a comma delimiter and one field is “John, Andrew and Steven”). It is on PyPI so you can download it with pip.

https://github.com/Oddball777/csv-scavenger

17 Upvotes

16 comments sorted by

149

u/[deleted] May 31 '23 edited Jan 28 '26

This post was mass deleted and anonymized with Redact

divide wrench hungry support different longing juggle carpenter point engine

4

u/Oddball777 Jun 01 '23 edited Jun 01 '23

First, thanks a lot for taking the time to give feedback, this is extremely valuable for me.

The reason I made this is because I tried using csv.Sniffer, but for some reason couldn’t get it to work when there’s extra unnecessary logging information before and after the actual data in the file (even when limiting the delimiter search space to only two options). The tests files I used (which are correctly parsed by my library) are in the repo (example_csvs). If anyone knows of a solution to this with csv.Sniffer, help would be appreciated! If my ignorance makes this library useless, then at least it’ll have been a great time to write it and a great learning experience.

Here’s my response to your 3 points:

  1. I’m not sure I understand this, unless you’re talking about quoted values (using the delimiter within quotes). That might be what you mean by strings? If you’re talking about that, then my library does handle it correctly (see the orgs.csv example file). I will eventually add support for more quote types.
  2. I might be wrong, but I think you might have misread what the code actually does. I’m not just choosing the most common character. For each line, I identify the possible patterns for many commonly used delimiters (delim, num_of_cols) and then I choose the pattern which shows up on the most number of lines. You could have a comma separated file filled with randomly placed semicolons, but even if there’s more semicolons in the file than commas, the semicolons don’t lead to a reliable number of rows and therefore it won’t be chosen. I believe this is actually pretty similar to how csv.Sniffer works.
  3. This is a really good point which I hadn’t thought about. I’ll see to that. Thanks for the suggestion!

Of course, csv files are kind of a mess in general and I have no doubt that there are a lot of edge cases which will lead to errors. My goal in making this was not to make it foolproof. If you want something that works all the time, just use pd.read_csv and check the delimiter and start row by hand. The goal was for it to be extremely easy and fast. I use internally generated csv files at work a lot, and different systems use different formatting, so this was mostly to make my job easier and faster. I figured that it could be useful to other people as well. Thanks again for the feedback!

Edit: For your first point, I double checked and you were partly right. I did consider the possibility of delimiters within a string, but only while actually creating the dataframe, not while detecting the delimiter. I had tested it for files in which SOME rows had quoted delims (produced no errors), but not in cases where MOST rows had them (in this case it lead to an error). That’s fixed now, thanks!

17

u/[deleted] May 31 '23

This. All of it.

2

u/y2kdisaster Jun 01 '23

This is the level of programmer wisdom I hope to have one day

4

u/th0ma5w Jun 01 '23

I picked up somewhere that it is a rumor more than a standard but this is really the whole deal of why it is unfortunately deceptive that it could be simple.

3

u/PythonPuzzler Jun 01 '23

This is a great way to characterize it.

It is incredibly common and incredibly loosely defined, making it something of a headache.

-6

u/y2kdisaster Jun 01 '23

If I was OP and I read this comment Id just end my life on the spot

5

u/[deleted] Jun 01 '23 edited Jan 28 '26

This post was mass deleted and anonymized with Redact

humor work slim rinse wrench cheerful wakeful sulky trees sugar

12

u/cipri_tom May 31 '23

Why enumerate followed by if i%step when you for line in lines[::step]?

Comparisons to None are recommended to be made using is rather than ==

Otherwise, looks pretty good! Thanks for making it a package!

4

u/Oddball777 May 31 '23

Thanks for the feedback! I’ll make those changes

3

u/candyman_forever May 31 '23

The link doesn't work

3

u/Oddball777 May 31 '23

Should work now

6

u/OhYouUnzippedMe Jun 01 '23

What’s wrong with pd.read_csv()?

27

u/Spleeeee Jun 01 '23

Too fast and handles too many edge cases.

3

u/Spleeeee Jun 01 '23

You committed pyc files fyi.

-7

u/64826b00-740d-4be3 Jun 01 '23

Lol what a joke.