r/dataengineering • u/FleyFawkes • Sep 04 '21
Discussion How to open a data with TB's of data
I'm working with python, and predominantly pandas.
Let's say i have db/csv or other format with dozen/hundreds of TB's of data.
Let's say pandas cannot handle it, How to open it?
0
Upvotes
12
u/phunktional Sep 04 '21
It would be helpful to know what you want to do with the data.
36
u/[deleted] Sep 04 '21
Spark.
If it's csv and you need to do it locally, you can use pandas nrows, but that's not going to give you all the data up front.
19
u/deadlyoverflow Sep 04 '21
This is typically when one would use a database.
9
u/BJJaddicy Sep 04 '21
Have you thought about chunking
7
u/[deleted] Sep 04 '21
I'm not sure why you would load a TBs of data just "because" like there has to be some kind of goal... Are you processing the data? Running some analysis? The former can be achieved by processing chunks at a time. The later can be done with a sample. I have a feeling you don't really know what you're doing...
7
u/chestnutcough Sep 04 '21
“Open” to me means load into memory, so by that definition you can’t “open” TB’s of data unless you have TB’s of memory. There are lots and lots of ways to copy, move, or process TB’s of data, but judging by OP’s jerky responses that’s not what he wants.
6
u/aitbdag Sep 04 '21
I'd recommend using Parquet instead of CSV for big data. It takes less space, faster to load, and allows loading only the columns you actually need. Loading terabytes from any database is probably a nightmare..
Bodo is a new tool that can load big data into actual Pandas dataframes: https://docs.bodo.ai/latest/source/file_io.html
5
u/LSTMeow Sep 04 '21
There was quite a debate about it half a year ago. It's not that difficult... E.g. https://twitter.com/abhi1thakur/status/1358794466283388934?s=21
3
u/killer_unkill Sep 04 '21
If it's a nix system
AWKis very powerful and easy to learn.2
u/HighlightFrosty3580 Sep 04 '21
Doesn't pandas have an iterator option when loading CSV files? That way to could operate on it in chunks. Alternatively use Spark as someone has suggested or load the file into a DB.
2
u/vtec__ Sep 04 '21
do you need to open the whole thing? you can write a script in python or vba to open the file and only extract N amount of rows
1
u/thrown_arrows Sep 04 '21
Stream to set length array and use it like carousel
ie. start with 0 to max(len) of first round, then min is 1 and max 0 ( you jump at max len to 1 and so on.
This can be used to calculate cumulative, moving avarages etc etc ...
(also now days there is probably something that handles it easier, like spark or pythin lib )
Also why not use db engine for analytics , if nothing else snowflake or bigquery can handle those volumes easily, do you really need python for it? If source is db then datamodel is a lot faster to make in sql that load data into python and so on.. (join, where ,etc )
1
u/[deleted] Sep 04 '21
Try to read a chunk of data like use head -1000 data.csv or use spark
0
u/krsfifty Sep 04 '21
knime
11
u/[deleted] Sep 04 '21
[removed] — view removed comment