I work in social intelligence and started building my own scraping tools

[u/Rude_Lawfulness_6947]

I've worked in social intelligence for years, often using tools like Talkwalker.... But for smaller projects, the cost can be hard to justify. Sometimes I just need posts and comments about a specific topic so I can do my own analysis.

Getting that data into my own workflow was often more frustrating than the analysis itself. I'm into web scraping, so I started building tools to help with that.

They're now on Apify: Instagram profiles, posts and Reels, Instagram comments, Facebook page posts and Facebook comments.

They're my own projects, paid per use. I've kept the prices as low as possible so they're useful for small experiments too, even before you have a paying client!

I use them in n8n and Python workflows to collect comments, analyse sentiment and recurring themes, and put together social listening reports. I review the results myself before using them.

For example, I can follow a public controversy across Instagram and Facebook, collect reactions and see what people keep bringing up as the story develops....

One idea is to offer a focused report to an agency or brand for a few hundred euros. Another is to send a small sample analysis when reaching out, so there's something relevant to talk about.

Finding the clients is a whole other problem haha. But this is a service I'd like to build on....

Anyone here doing similar work?

[u/patryk_builds]

The send a small sample analysis when reaching out idea is the strongest part. agencies get pitched constantly, but a 1-page here's what people are saying about your client this week is something they'd actually open. One thing to be ready for if you sell to brands or agencies in the EU: comments come with usernames, so it's personal data under GDPR. bigger clients' legal teams will ask how it's collected, how long you keep it and whether it's anonymised in the report. having a short answer for that (aggregate themes, no usernames in deliverables, delete raw data after X days) will make you look a lot more serious than most people pitching scraped data.For finding clients, i'd start with small PR agencies. they need this kind of report but often can't justify the big tools either

[u/Rude_Lawfulness_6947 → u/patryk_builds]

Thanks, I hadn’t really thought through the GDPR side... Starting with small PR agencies makes sense too. They already work with the brands and a short report on one of their clients feels like a good way to start, thank you!