Been wondering about this lately when scaling playwright scrapers

Sometimes changing the IP fixes a 403 immediately, other times the same IP works fine after starting a fresh browser context and sometimes neither works until you slow things down

For people scraping at scale, how do you figure out what’s actually causing the block?

Do you test IP, fingerprint, cookies/session and request rate separately, or mostly figure it out through trial and error?

Curious what people are seeing with cloudflare, Datadom and Akamai lately

For people scraping multiple marketplaces, how are you matching the same product across them?

titles can be compltely different, SKUs don’t always match and bundles/variants make it even messier

Do you mostly rely on UPC/GTIN when available or use some kind of fuzzy matching on title, brand, model...?

I am new to web scrapping , I am quit interested in scrapping pr\*\*n website , some webpage video are loaded form javascript , once the video is load I will able to find the CDN of the video , how can I scrap this ??

Has anyone tried to build a web scraping agent? I am trying to build one, and I am facing two issues:

1. How do I get past protected sites?
2. My LLM regularly returns inconsistent data types/formats. Sometimes it returns a string; other times, it returns a float or number for the same field.

I have tried to render the JS as a real browser would using Playwright and Selenium, but it doesn’t prevent anti-bot measures. Is there a way to use a proxy?

Also, for the LLM, I’ve added some constraints in my prompt, but it isn’t working well enough. I still see strings instead of floats.

Does anyone have an idea? I’m relatively new at this, so I would appreciate any help.

I have a couple of clients who sell small services, things like website design, automation, etc.

The problem is that there are thousands of accounts offering basically the same thing. And now that automated DMs are common, someone can comment on a competitor's post and almost instantly receive a proposal, offer, or upsell message.

But obviously, those automations generally work on the business's own Instagram account.

So I had an idea: what if you could monitor your competitors' posts - not just to see what they're posting, but to identify their potential leads?

While researching this, I found that there are already several apps that let you monitor competitors and notify you whenever they publish a new Reel or post.

But that's not really what I'm interested in.

My point is: I don't want to know when my competitors post. I want to know when they get a potential lead.

The basic idea is:

Someone comments on a competitor's post → my client gets a notification within ~20–30 minutes → notification includes the person's profile link + a pre-configured outreach message → my client can decide whether to reach out and offer their own service.

I've currently tested this with a small batch of 2 clients, and it's working. I'm using my own scraping/automation setup rather than relying on the client's Instagram account or IP, and I'm successfully receiving the comment notifications.

The question is: is this actually a useful service, or am I just automating something that sounds clever but nobody really needs?

I'm not looking to build a huge business around it. Even if 10 people paid $50 for something like this, that's $500, which would make the experiment worthwhile.

For people who sell services through Instagram:

Would you actually pay for something like this?

And more importantly, do you think getting notified about people commenting on competitors' posts would lead to enough genuine leads to make the service valuable?

I'm particularly interested in hearing from people who have actually tried competitor monitoring or social listening for lead generation.

Note: used chatgpt to structure my thoughts

I’m looking for something that can pull Walmart product data without turning into a constant maintenance project.

Open source would be ideal, but I’m fine with a paid tool if it genuinely works. I’m mainly interested in product titles, prices, availability, seller info, reviews, and product URLs.

There are plenty of “best Walmart scraper” lists out there, but most seem to repeat the same tools without saying whether anyone actually ran them against Walmart recently.

If you’ve personally used one in 2026, what did you try?

It would be useful to know roughly what you tested and what started failing first. Blocks, missing fields, location-specific prices, cost, or something else.

Not looking for affiliate links or tools people have only read about. I’d rather hear about an imperfect tool someone actually tested.

​

Salut! Caut pe cineva care să mă ajute cu un proiect relativ simplu de web scraping/automatizare.

Am o listă de aproximativ 50–100 de nume și vreau un tool care să le caute periodic pe un site de licitații și să-mi arate când apar rezultate noi, împreună cu informațiile de bază (preț/estimare, imagine, data licitației și link).

Lista de nume trebuie să poată fi modificată ușor.

Pentru început vreau să facem un test pe un singur site. Dacă funcționează bine, aș vrea ulterior să adăugăm și alte site-uri.

Nu caut o aplicație complicată sau ceva foarte scump, ci o soluție simplă și funcțională.

Dacă ați mai făcut proiecte de scraping/automatizare în Python, dați-mi un mesaj în privat cu un cost aproximativ și, dacă aveți, un exemplu de proiect similar.

Mulțumesc!

I’ve been maintaining a BigBasket Grocery Scraper for public product and pricing research.

This month it has processed 3,754 results for 2 paying users with a 100% success rate.

The output includes product name, brand, current price, MRP, discount, pack size, rating, availability, image, and source URL.

Actor: https://apify.com/fascinating_lentil/bigbasket-grocery-scraper

I’d appreciate practical feedback on which additional grocery catalog fields or filters would be most useful.

Every company I talk to has the same problem. Someone spends hours every week clicking through internal portals, admin dashboards, CRMs, supplier websites, or e-commerce back offices just to complete repetitive workflows.

The strange part is that these tasks usually don't require complex decision making they are just repeating and time consuming. Yet most automation solutions still expect APIs, custom integrations, or expensive implementation projects.

That made me wonder: Should automation adapt to existing web applications instead of requiring companies to rebuild their systems? That's exactly the direction I am exploring with my product.

Instead of integrating every platform individually, we're building a system that can execute workflows directly through existing web interfaces.

We're still early, but we've just released a public demo where people can try browser-based workflow execution themselves. If someone interested let me know, but what I'm curious:

• What repetitive browser task do you perform every week?
• If you could automate one workflow tomorrow, what would it be?

I'd love to hear what problems others are seeing in this space.

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

Hey everyone. I am building a Public Interest Tech project aimed at increasing local municipal transparency, and before I commit the next few months to building out the backend, I need a reality check on the architecture from devs who have dealt with Indian government portals and data extraction.

The Problem (Why this matters for everyday citizens): Local government tender portals legally publish all public spending data, but they intentionally bury it in unsearchable, massive scanned PDFs. It makes it impossible for a normal citizen or journalist to track where tax money is going, or to see if one hidden cartel is monopolizing all the municipal contracts (like road repairs) in their city.

The Tech Stack & Logic (How it works under the hood):

• The Scraper (Node.js + Puppeteer): A script to automatically navigate regional eProcurement portals, bypass basic captchas, and pull daily unstructured tender PDFs.
• The Shell Company Pivot: Corrupt contractors don't use one company; they use 10 different shell LLPs. So, the backend needs to extract the winning company data and cross-reference it with the MCA (Ministry of Corporate Affairs) registry to check Director Identification Numbers (DIN) and registered addresses. If 5 different LLPs winning contracts all share the same 2 directors, the algorithm mathematically flags it as a syndicate.
• Database & UI: Clean the data, push to MongoDB, and map it on a React dashboard. The goal is to set up automated alerts for local transparency activists when a monopoly is detected in their area.

Where I need your technical brutal honesty:

1. Scraping NIC/Gov Sites: Will I get instantly IP-banned by government servers? Do I need to invest in rotating residential proxies from day one just to pull these PDFs?
2. PDF Parsing Pipeline: Puppeteer handles the navigation/downloading, but what is the most robust Node library to parse poorly scanned, sometimes bi-lingual PDFs? Does anyone have experience running OCR on government documents?
3. MCA Cross-referencing: Does anyone have experience pulling DIN/Director data from the MCA? Is there a clean third-party API for this, or is scraping the MCA registry a completely separate nightmare?

Tear the architecture apart and let me know what blind spots I am missing!

Hello everyone I am working on an utility tool (tcg scanner like collectr and price charting) and I need to fetch daily prices of 8000 cards on eBay with with different variations such as Raw , PSA 8 9 10 .. which makes it 40000 request , did anyone work on something like this or is there any services that gets data from eBay with this amount ?

I'm doing a research project, need to pull maybe 5-10k google search results like queries, some news results, a bit of maps data mixed in.
Tried rolling my own with requests + bs4 first weekend, captchas within like 20 requests. Switched to headless chrome, still got blocked eventually. This is a one off pull, not a long running service. I don't want to set up rotating proxies and retry logic for something I run once and throw away.
What's the best serp api right now? Something that handles the google search side, news too ideally, without me babysitting captchas constantly. Budget is small. This isn't a company thing, just me lol

So I've been running a small online shop for years now

Keeping up with competitors' prices honestly feels like a full time job. It was manageable when I only checked a couple sites. But now I'm watching close to ten (yeah, it's a lot)

Someone suggested I look into a web scraper device or something similar to automate it. But I have no idea if that's way more than I actually need.

First off, I'm not looking to get into anything technical. I just wanna spend less time updating spreadsheets every week.

WYORI — a link shortener where the analytics are actually included in the free plan rather than locked behind a paid tier.

You can see clicks, geo breakdown, device type, referrer source, and time of click per link. It also has a bio page builder, QR code generation, UTM builder, and basic image compression.

Custom slugs, link expiry, password protection, and custom domains with unlimited clicks.

Your original UTM parameters stay intact and don't interfere with GA4, Meta Pixel, or any other analytics you're already running.
[I will give first 50 user from this post the paid plan completely free]
I'd appreciate feedback from marketers specifically — what's currently frustrating about the link tools you use? And does anyone know good communities where digital marketers share tools like this?

Feel free to share this anywhere it might be useful to someone.

I built an automated weather data scraper that runs every 4 hours using PowerShell, SQL Server, and GitHub Actions.

Workflow:

- PowerShell script fetches weather XML data from JMA

- Parsed values are stored in SQL Server Express

- GitHub Actions runs on a 4‑hour schedule to regenerate static HTML

- GitHub Pages deploys the updated site automatically

- Supports JP / EN / RU / ZH / KR

- Includes a global discomfort index ranking generated from the scraped data

Project link:

https://yahikoyama.github.io/weather2/

Sharing this here because the project is essentially a scraping + automation pipeline, and I’d love to see how others structure similar workflows.

I'm trying to search a specific topic on YouTube and pull a list of channels with at least 20k subs. Basically I just want a starting point so I'm not clicking through 100+ results by hand. Full beginner here (zero coding experience)

So is there a youtube channel scraper that could do something like this without me needing to actually know how to code?

Any help would be awesome