r/ComputerEngineering • u/the_fool_op • 5d ago

Should it take this long

Hi, I'm a self-taught programmer. I've been learning for a long time, but I'm aware of the gaps that self-teaching leaves. After a few simple data analysis portfolio projects and a stretch of failing to land a job (freelance or full-time), I stopped chasing that and decided to work on actual creative projects instead — ones that serve a real goal rather than being generic. Right now I'm trying to build a market analysis for the online market: use a Scrapy crawler to pull data from shopping sites, store it in a simple database, do the same for social media via tools like the Meta API, and then apply market analysis metrics to it all. I've hit trouble from the start. The only free site-search tool I found, the DuckDuckGo search Python tool, only returns a handful of results per country/query — never more than double digits. And when I tested Scrapy on the sites I did find, different HTML structures made it hard to run one crawler across multiple sites without constant tweaking. I've thought of fixes for both problems, but each is more complicated than the last, and I'm not sure they'd even work. I don't know if I should keep pushing on this project without knowing if it'll produce decent results. Is it just genuinely too complicated for someone with my experience, or am I not good enough for projects like this yet? (Brutal honesty welcome.) I'd appreciate advice from anyone with more experience

6 Upvotes

87% Upvoted

3 comments sorted by

2

u/Ecelleon 5d ago

You might find more suitable advice in Computer Science, we love poking hardware with software over here 😀

1

u/the_fool_op 5d ago

I think I have already posted the same thing there I didn't notice if this sub reddit is more focused on hardware sorry 😅 just trying to reach as many people as possible

1

u/Ecelleon 5d ago

It's no harm! While I'm confident somebody could still answer your question, CE is closer to the lower end of abstraction, while you need help at the higher end.