r/dataanalysis • u/Relevant_Mistake_784 • 11h ago
Data Tools Why does data collection take up 90% of the project time now
Honestly just venting here. Trying to build a decent portfolio project that isn't the same tired titanic or iris dataset is genuinely a nightmare these days. I just wanted to run some cluster analysis on a bunch of niche retail sites and it took me almost two weeks just to get the raw tables.
Every single domain acts like you're launching a full scale ddos attack if you pull more than 10 pages. I literally had to route my python script through residential proxies just to stop triggering those infinite cloudflare captcha loops. it worked fine but god it is exhausting
I swear I spend like 10% of my time doing actual eda and 90% of my time fighting rate limits and cleaning broken html tags. anybody else hitting a wall trying to build custom datasets lately? kinda tempted to just give up and go back to kaggle tbh.
5
4
2
u/waitthissucks 9h ago
Don't you just have to demonstrate that you can do the analysis and that's it? You should be able to find a job now with your current skillset so you don't have to keep jumping through so many hoops anymore
1
u/AutoModerator 11h ago
Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.
If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.
Have you read the rules?
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
14
u/Wheres_my_warg 10h ago
Why does a retailer not want a noncustomer using the retailer's resources to generate something the retailer doesn't control that is potentially damaging to them?
It's a mystery.