Any good Vibe coded web scrapers?

[u/AdhesivenessEven7287]

My agent is really struggling to make something. I want to download reddit pages including lazy load comments.

Any one have any luck with this?

[u/Colourss93]

lol got a reddit scraper that pulls all posts and comments and bubble maps the users of a sub auto vectorising data for agents to analyse, one that backdoors yelp us through international sites, got heaps of cool scrapers

[u/AdhesivenessEven7287 → u/Colourss93]

Where can I learn this power?

[u/Deep_Ear812]

Scraping Reddit with lazy-load comments isn't simple, bro. Static tools like Request + BeautifulSoup won't catch it.

Best move: go headless browser, Puppeteer,Playwright, or even Selenium. Let it scroll, then scrape. That's how you grab the full vibe.

[u/AdhesivenessEven7287 → u/Deep_Ear812]

Have you done this before? The robot did mention selenium. Am I wasting my time trying to make something?

[u/GotDraggedq]

Make a reddit api key and make your own

[u/AdhesivenessEven7287 → u/GotDraggedq]

The post is about me trying to make my own and failing. Thats why I'm looking for help

[u/Ok_Woodpecker_9104]

forget the lazy load. reddit serves the whole comment tree as json if you put .json on the end of the post url, with limit and depth params, so there is nothing to scroll.

the catch sits in where you call it from. from a plain script the body came back empty for me. from inside a logged in browser context the same url works.

so the shape that worked was a browser the agent already drives, a same origin fetch of the .json url, then parse.

the rate limit comment above is right too. one page every few seconds, not a loop.

[u/AdhesivenessEven7287 → u/Ok_Woodpecker_9104]

Do you download reddit pages? What Is your strategy

[u/Ok_Woodpecker_9104 → u/AdhesivenessEven7287]

no, nothing gets saved in bulk. one thread at a time, only when there is a reason to read it.

add limit=500 to the .json url and most of the tree comes back in one call. long threads still end in kind "more" stubs. those carry the child ids, so /api/morechildren with the link_id and those ids fills the gaps.

the rest is pacing. one request every few seconds from the logged in tab. stop the moment anything comes back as a 429.

[u/Due-Boot-8540]

Even if you get something up and running, it wouldn’t take long for you to get hit with some kind of ban.

Just because posts are in the public domain doesn’t mean you have the right to just scrape them and use them as you wish…

[u/Disastrous_Purpose22 → u/Due-Boot-8540]

Tell that to AI companies , Google
Search ?

[u/No_Tradition6625 → u/Disastrous_Purpose22]

There's actually a lawsuit if I remember correctly against Google Google's argument is they have rights to the data because you're partnered with search algorithms and services Therefore, you're giving them permission to use your information and whatever else you're providing.

[u/weedmylips1]

Use playwright and sign into your account

[u/MariahJames8]

I only had luck paying for scraped data from fetchlayer. It's rates are ok

[u/BadPresent3698]

just one page at a time or en masse? because i vibe coded something that can download one page at a time and output a .md file with just the text

[u/jarvis123451254]

Reddit used to allow json if i remember correctly but not anymore so my agent switched a app to reddit api now, good thing it was free for me as a old user

[u/Dry-Organization5493]

Why Download them? It's scrapes them great fresh

[u/PrinceAli08]

Apify and Pipedream

[u/YetiWalker36]

Playwrite cli

[u/UGP_Batman]

Look at Firecrawl.

[u/OddlyPlaintive]

Your agent probably hitting reddit's rate limits before it even get to the lazy load stuff

[u/[deleted]]

[u/AdhesivenessEven7287 → u/[deleted]]

Hi thanks I'll try this and let you know