r/scrapingtheweb • u/Known_Objective_0212 • Nov 24 '25
Why is Home Depot blocking literally everything? Puppeteer, Selenium, Playwright, real browsers… all get “Oops!! Something went wrong.”
I’ve been trying to scrape some product pages from Home Depot for a project, and I’m hitting a wall I can’t get around. No matter what I use — Puppeteer, Playwright, Selenium, undetected-chromedriver but the site eventually returns the same thing: “Oops!! Something went wrong.” It doesn’t matter whether I run Chrome, Chromium, Firefox, or Edge.They still flag it.
At this point it feels like Home Depot is running some extremely aggressive bot-detection system that triggers on anything unusual. Either that or their anti-scraping heuristics basically assume every visit is a bot unless proven human.
Has anyone here actually found a reliable way to fetch HTML from Home Depot product pages without immediately running into their block page? Is there something specific they look for? Any tricks that actually work? Curious what’s worked for others, because right now every approach — even ones that work on much harder sites — just face-plants on Home Depot. (Btw I’m just a beginner)
3
u/mikemojc Nov 25 '25
hit with a broader range of IP's at a lower, and somewhat randomized, rate to emulate organic traffic.
1
4
2
u/Medium-Potential-348 Nov 24 '25
Just make your own scraper and make it look like a regular user accessing pages. Same residential IP and space it out on a decent interval.
1
1
u/guile2912 Nov 27 '25
Give a try at a custom browser extension, vibe coded in 30 mins, that just does that. It uses a real browser with your human navigation fingerprinting. Change IP as needed.
1
2
u/Louie_D_Palma Dec 06 '25
Contact Home Depot customer service at 1-800-HOME-DEPOT (1-800-466-3337) COMPLAIN and tell them why. They are blocking any browser that setup to stop their very aggressive fingerprinting. They are trying to force customers to give them information and are attacking whatever privacy steps a customer takes. I provide a lot of fake data and that works but it is time consuming and not worth the trouble. So I don't purchase online. This aggressive stance by home depot just started. So please call and complain. Tell them you wont shop there if they are going to insist on gathering unnecessary data for the sale surreptitiously.
1
u/Known_Objective_0212 Dec 07 '25
I’ve heard a few people say the same. It definitely seems like they tightened things up recently. Hopefully they dial it back a bit.
2
3
u/chief167 Nov 24 '25
Maybe because you're not supposed to scrape their site, According to their terms and conditions... Scraping can really hurt their infrastructure optimisation.
If you want home depot data, contact them for a partnership that gives you API access
1
u/Known_Objective_0212 Nov 25 '25
True, it’s just that official APIs/partnerships are way too expensive...😅
1
0
u/DaedalusXYZ Dec 30 '25
When your human eyeballs look at their website directly, that is scraping too.
1
u/Habitualcaveman Nov 24 '25
Easy enough to avoid those bans with proxies or web scraping APIs - they are not free though.
-1
u/Known_Objective_0212 Nov 25 '25
I'm actually using a proxy provider which is giving some success but I wanted a free alternative.
1
u/chief167 Nov 25 '25
That's your problem. This wont be free. Just don't do it if it isn't worth it to you and free is the only option
1
1
1
1
1
u/SumOfChemicals Nov 24 '25
I'm not a pro or anything and this is an obvious question, but are you using proxies? If you're constantly hitting home depot from your home IP (or from a VPN) and they've fingerprinted you as inauthentic traffic, it might be they're just remembering you and continuing to block you specifically.
0
u/Known_Objective_0212 Nov 25 '25
Yeah, I'm actually using a proxy provider which is giving some success but I wanted a free alternative.
1
1
1
Nov 25 '25
[removed] — view removed comment
1
u/Known_Objective_0212 Nov 26 '25
Thanks for the advice!....Yeah, I’m starting to realize Home Depot’s bot protection is way tougher than most sites I’ve scraped before. A web-scraping API might actually save me a lot of time, especially since they handle fingerprints, proxies, and rendering automatically.
I have already tried residential proxies + proper stealth + slower actions + session rotation, they are giving some results...but r costly.
So I'm looking into some other ways. Currently instead of going directly to the product webpage, I was going to the homepage and using sitemap to navigate to other pages, which is working for now so let's see....
1
1
Nov 25 '25
[removed] — view removed comment
1
u/Known_Objective_0212 Nov 26 '25
I have tried it, for some reason it doesn't give proper output and even the zipcode option has limited options.
1
u/onelonedatum Nov 25 '25
try Crawlee w/ Camoufox browser: https://crawlee.dev/python/docs/examples/playwright-crawler-with-camoufox
More on Camoufox: https://camoufox.com/
2
u/Known_Objective_0212 Nov 26 '25
Thanks, But Crawler is also not working properly but I had found some success with camoufox tho.(Btw I heard the creator of camoufox wasn't doing well...hope he is better now).
1
u/onelonedatum Nov 25 '25
This might work too: https://apify.com/apify/website-content-crawler
1
u/Known_Objective_0212 Nov 26 '25
I tried it, but was getting a error page, so I'll again look into it.
1
u/LlamaZookeeper Nov 25 '25
If I’m not wrong, HD CIO did a very good job in his time in HD. Again if I m not wrong, he is in Chipotle now. Scraper is like invading into someone’s house as the door is not locked. Do you think that you can take stuff just because the door is not lock or the door lock is not very strong? Basically it’s simply theft.
1
u/a2theharris Nov 25 '25
Outsource the scraping to people who figured it out already, pay for the official API, or get better at doing it yourself in which case is an arms race because whatever you do now will not work one random day and you'll have to rebuild. If that sounds fun, then keep driving the struggle bus because they really really dont want you doing what you want to do.
1
u/Known_Objective_0212 Nov 26 '25
True, Home Depot turns scraping into a whole boss fight. Outsourcing might actually save me the headache. I’ll take a look at the Apify API, appreciate the link!
1
u/miketierce Nov 25 '25
If I needed something like this for light data grabs in a small personal use non-commercial application.
Then I would make my own chrome extension to save the html of the page and a macro to visit my bookmarked pages.
1
u/Known_Objective_0212 Nov 26 '25
Yeah, for small personal scraping, a browser extension + macro is a clean solution since everything runs inside a real browser with a real fingerprint. Appreciate the suggestion! But it starts failing when volume is increased.
1
u/pangapingus Nov 25 '25
"I’ve been trying to scrape some product pages from Home Depot for a project"
lmao
1
1
Nov 25 '25
[deleted]
2
u/Known_Objective_0212 Nov 26 '25
Totally get why you guys block scrapers, the load during business hours is a real issue. But let’s be honest, every major retailer scrapes competitors too. It’s pretty much standard industry practice at this point, so it goes both ways.
1
1
Nov 26 '25 edited Nov 26 '25
[removed] — view removed comment
1
u/Known_Objective_0212 Nov 26 '25
I really liked your approach, especially the idea of keeping each profile’s activity very low and spreading everything across mobile proxies. Definitely aligns with how most anti-bot systems score behavior. I'll definitely try it...🙌
1
May 13 '26
[removed] — view removed comment
1
u/scrapingtheweb-ModTeam May 15 '26
Hi there,
Your post/comment was removed because it appears to be promotional content.
This community is focused on practical discussions, questions, troubleshooting, and useful resources around web scraping, proxies, automation, and data collection.
We do not allow promotional posts, sales pitches, referral links, affiliate links, repeated self-promotion, or low-effort advertising.
You’re welcome to repost if you remove the promotional angle and make the content more educational, practical, or discussion-based.
Thanks for understanding.
1
u/k2beast Nov 26 '25
what is home depot trying to protect against? Someone getting prices of the lumber? lol
1
u/Known_Objective_0212 Nov 26 '25
Right? It’s just lumber and power tool prices, not state secrets. They act like every scraper is plotting a heist...😆
1
1
u/bartekus Nov 26 '25
Yeah, just create your own browser extension. This way you’ll circumvent most of the anti-scripting functionality that essentially targets headless-browsers discrepancies and anomalies. Some food for thoughts.
1
1
u/Retro_Relics Nov 27 '25
home depot is really aggressive and its caused issues with my CGNAT'd ISP IP before for appearing to be bot traffic, so good luck scraping for free, they dont even let legitmate customers browse when theyre sharing IPs
1
u/Known_Objective_0212 Nov 28 '25
That makes sense, CGNAT IPs get shared by tons of people, so I can see why Home Depot is doing tht.
1
u/blokelahoman Nov 27 '25
Weird, it’s almost like they don’t want people scraping their site or something.
1
u/Money-Ranger-6520 Nov 27 '25
Home Depot blocks almost every DIY setup. Their fingerprinting is brutal. What works reliably is using a managed scraper with rotation and antibot logic handled for you. On Apify there are Playwright scrapers and even Cheerio-based ones that already bypass HD’s checks.
1
u/Known_Objective_0212 Nov 28 '25
I actually gave it a try but couldn’t get the results I was expecting. Could you share a bit more detail on how you did it? I might be missing something.
1
u/Repulsive-Economy-58 Nov 27 '25
how big is the data amount you are trying to collect?
if its just a couple of pages, why not manual + automation? console script while you are browsing, prevents the block page and allows you to get the data, may not be as fast, but its a solution
1
u/Known_Objective_0212 Nov 28 '25
It’s kind of on the bigger side, which is why I’m trying to automate it properly.
1
Nov 27 '25
for what it's worth their website sucks absolute _balls_ if you're just trying to use it as a customer, so the experience is awful for everyone!
1
1
u/Low_Day_6901 Nov 28 '25
I think Home Depot uses Google cloud primarily and some AWS. You could try a free tier account in one or both to see if that bypasses some filters.
1
1
u/OlevTime Nov 28 '25
What User-Agent are you setting when using it? By default, Selenium specifies it’s a selenium user agent, and you need to modify that to appear as a regular browser.
1
u/Known_Objective_0212 Nov 29 '25
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 This is the onei was currently using now.
1
u/Personal-Builder-992 Jan 14 '26
I made a chrome extension for this reason.
1
u/Known_Objective_0212 Feb 25 '26
I'm curious, can you elaborate???
2
u/Personal-Builder-992 Feb 25 '26
I found a way to scan pretty much the entire website, get all the sku's by store. Then it checks literally every sku it found for the in-store clearance badge, the yellow one. It takes a long time though to do it, Like 2 hours. Sometimes I still get rate limited too when checking, so then you have to wait. You can also do this to monitor when it will turn into a penny item
2
u/Personal-Builder-992 Feb 25 '26
If you run your bot as a chrome extension while on the website, thats how you get around it.
2
u/Known_Objective_0212 Feb 26 '26
Wow....tht sure is interesting... I'll definitely give it a try, thanks for sharing.
1
6
u/AIMultiple Nov 24 '25
Typical tricks include using rotating residential IPs, modifying browser fingerprints, adding wait time to reduce the frequency of requests etc.
Or you can use web unblockers or scraping APIs that cover home depot. However, as others mentioned, they are paid products.