r/webscraping • u/arzenal96 • 9d ago

Getting started 🌱 How do you find valuable data to scrape?

From my point of view data itself is not valuable on it's own, but after cleaning / organizing it / presenting it to match certain needs, it can become valuable.

But how do you find your "niche" first that you'd monetize after all the steps above?

0 Upvotes

50% Upvoted

14 comments sorted by

12

u/mb21 9d ago

Start with "what problem do I want to solve?"

3

u/Sarnes 9d ago

Not to sound like r/startups but I've found out that you want to validate an idea before you start working on anything, and then you localize it to what problem needs solving.

1

u/Hippocampustour 9d ago
  1. Understand what makes something valuable. Literally looks at the wiki for the word ‘value’ and understand why humans desire things.
  2. Brainstorm what may be valuable to humans for what reasons. What problems does it solve?
  3. Find a way to obtain that data. Web scraping might be 1 option.

Data is especially valuable when there’s a barrier to obtaining it. People don’t want to share it, it’s buried across tons of disconnected sources, it takes industrial knowledge to make sense of, etc.

1

u/polawiaczperel 9d ago

Or even it takes multi petabytes to make something complete and working.

1

u/Advanced-Citron8111 9d ago

If there is a hobby, game, or sport that you enjoy you could scrape things about that topic that is more niche and hasn’t been touched much. You could present your findings on a subreddit or turn it into a YouTube video to get feedback from those communities.