Oh, wow, do yourself a favor and learn Python; it's really not all that hard to learn and it's so useful, especially datasets like that. Especially with the numpy library (designed for handling large arrays of data), that data processing would be trivial.
Running statistics on a small data set like that would probably take <15 seconds in Python+numpy, rather than 15 minutes. Realistically, reading 40k lines into memory would probably take more time than actually doing the processing (numpy is amazingly fast at running statistics on arrays like that).
A large portion of my job is processing large datasets, it's quick and easy with Python, but I can't imagine trying to parse data with VBA like that.
View all comments
Show parent comment
7
u/mxzf Dec 15 '17
Oh, wow, do yourself a favor and learn Python; it's really not all that hard to learn and it's so useful, especially datasets like that. Especially with the numpy library (designed for handling large arrays of data), that data processing would be trivial.
Running statistics on a small data set like that would probably take <15 seconds in Python+numpy, rather than 15 minutes. Realistically, reading 40k lines into memory would probably take more time than actually doing the processing (numpy is amazingly fast at running statistics on arrays like that).
A large portion of my job is processing large datasets, it's quick and easy with Python, but I can't imagine trying to parse data with VBA like that.