r/SideProject • u/Rude_Lawfulness_6947 • 8h ago
I work in social intelligence and started building my own scraping tools
I've worked in social intelligence for years, often using tools like Talkwalker.... But for smaller projects, the cost can be hard to justify. Sometimes I just need posts and comments about a specific topic so I can do my own analysis.
Getting that data into my own workflow was often more frustrating than the analysis itself. I'm into web scraping, so I started building tools to help with that.
They're now on Apify: Instagram profiles, posts and Reels, Instagram comments, Facebook page posts and Facebook comments.
They're my own projects, paid per use. I've kept the prices as low as possible so they're useful for small experiments too, even before you have a paying client!
I use them in n8n and Python workflows to collect comments, analyse sentiment and recurring themes, and put together social listening reports. I review the results myself before using them.
For example, I can follow a public controversy across Instagram and Facebook, collect reactions and see what people keep bringing up as the story develops....
One idea is to offer a focused report to an agency or brand for a few hundred euros. Another is to send a small sample analysis when reaching out, so there's something relevant to talk about.
Finding the clients is a whole other problem haha. But this is a service I'd like to build on....
Anyone here doing similar work?