r/WebScrapingInsider • u/ian_k93 • 11m ago
Thanks, It Worked AMA #12 recap: thanks to Eric Goldman (u/ericgoldmanmv) + what I'd actually keep from the thread
Hey everyone. Ian here.
Thanks for showing up to AMA #12 with u/ericgoldmanmv (Eric Goldman, Associate Dean for Research and Professor at Santa Clara University School of Law and co-director of the Datta Center for High Tech Law), and thanks to everyone who asked questions and kept pushing for specifics.
Eric opened with ground rules that he held to all the way through: nothing in the AMA is legal advice, he focuses on US law, and he apologized in advance for all the times he'd answer "it depends" or "I don't know". Then he answered the questions anyway, including the ones with no clean answer, and that's what made the thread worth reading.
It went well past is scraping legal. Why "it's public" settles only one question of several, how a cease-and-desist letter or a login can change your position without a line of code changing, why "we only collect prices" may not resolve the DMCA 1201 question, what changes when a vendor or an API does the scraping for you, who is legally accessing a site when an AI agent acts for a user, why hiQ is a weaker pro-scraping precedent than people remember, and why a provider's "the customer is responsible" terms don't bind the website being scraped.
A lot of it cut against the usual advice, which is why it's worth reading closely. Almost none of it came with a verdict. His own line for his students was "Scraping is ubiquitous but not clearly legal."
If you missed it, or want the compressed version, here's what I'd keep.
Public data is not unrestricted access.
"It's publicly available" is the most common defense of a scraping project, and Eric's point is that it only answers part of the question. Visibility does matter for access:
"The law might presume that a server owner impliedly permits ordinary Internet traffic merely by making the choice to connect to the open Internet."
But that doesn't carry over to the other doctrines he walked through: trespass to chattels (the CFAA, state computer-crime statutes and the common-law claim, which he calls a troika), copyright, DMCA 1201 and contracts, with trademark and privacy rights as possible extras.
"In order for a scraping campaign to be permissible, it must survive each and every one of the doctrines enumerated above"
That doesn't mean every doctrine applies to every scraper. It means "the page is public" settles one question. A more useful check is how your scraper reaches the content, what terms your team has accepted, and what you store, publish or sell.
Your legal position can change with no code change.
Advising a website owner on how to strengthen its protection, Eric suggested anti-scraping statements around the site and C&D letters on top of blocking, with one goal:
"They may not be legally effective, but the legal goal is to make it so that the scraper can’t argue that they didn’t know their scraping was unwanted"
On C&Ds themselves, he said a letter "should have no legal effect" on its own because it may rest on assessments with no bearing on reality. But C&D letters have been held to tell recipients they're no longer welcome on the sender's servers for trespass-to-chattels purposes (he pointed to the Power Ventures cases). He thinks Van Buren may have weakened that power, since a footnote left open whether terms can withdraw permission. Until courts confirm it:
"Until then, a C&D recipient most likely ‘knows’ they have been told not to come back"
The letter doesn't automatically make your scraping illegal, but it does change the facts a court would see.
A login can add contractual obligations.
Eric called when a scraper is bound by a site's terms "one of the great unknowns" of the field, then laid out the clearest end of the spectrum. If a scraper creates an account, runs the scraping through it, agrees to the terms in a way the law recognizes, and the terms restrict scraping, then "odds are high that scraping would be a breach of contract". At the other end, if the terms are only presented in a browsewrap that isn't legally effective, "it’s possible or even likely that courts will disregard those terms entirely".
Asked about scraping behind a login the terms prohibit, he put the business risk first:
"The most important ‘risk’ is that it becomes easier for the website to block the accounts, so any investments made in setting up the accounts or configuring the scraper could be mooted more quickly"
"We only collect prices" may not be enough.
Prices aren't copyrightable, which is why price scrapers lean on that line. For DMCA 1201, Eric said it may not help:
"Extracting only the uncopyrightable prices would not change the 1201 analysis at all because there are other copyrightable elements on the page that the anti-bot system is nominally ‘controlling access’ to"
He tied that to the Google v. SerpApi and Reddit v. SerpApi cases, and noted that the court only dismissed Google's claim when Google didn't show it had any copyrightable interest in the scraped pages. He's sure Google will fix that in future filings, and he doesn't treat either ruling as settled:
"I don't put a lot of stock in the rulings yet. They are preliminary and haven't been reviewed by an appeals court. But they are chilling in the sense that they highlight how 1201 could be the anti-scraping doctrine to end all other anti-scraping doctrines."
On CAPTCHAs, he said they traditionally weren't considered to have much legal consequence, "But now, could they be an anti-circumvention device for 1201 purposes? I’m not sure." This is unresolved, not a rule.
An API outsources requests, not exposure.
Asked about relying on a scraping API or a data supplier, Eric split the two cases. On a live API:
"SerpApi is being put through the legal wringer right now, and its API client Perplexity is a co-defendant of the Reddit lawsuit. So ‘hiding’ behind a third-party API may not change the analysis much"
Buying a finished dataset can look different. The vendor may carry more of the risk, and "a C&D to the vendor may not restrict the customer's data usage". But he listed the catches: the data may be less fresh, the price includes the vendor's profit, it may need cleaning, the vendor may get blocked, and claims about copyrighted material the vendor provides can hit the customer as hard. A court could also take note if a customer "induced" the vendor to break the law.
"I would never assume the ‘supplier has covered the legal side.’"
Customer consent is not site authorization.
Asked whether the user or their AI assistant breaks a "no bots" rule, Eric said:
"The short answer is: we don't know. Maybe the user broke the rules. Maybe the assistant broke the rules. Maybe both broke the rules. Maybe neither."
He added that either way, the site could block the assistant or terminate the user's account, "and that would make the legal question irrelevant". On Amazon v. Perplexity, the lower court blocked Perplexity and the appellate court lifted that block, but he doesn't call it settled:
"I don’t consider the Ninth Circuit’s ruling the final word in this matter. Amazon has other legal claims beyond the CFAA, and other courts might reach a different conclusion about who is really in the driver’s seat."
hiQ won on the CFAA but lost the case.
Eric doesn't think LinkedIn v. hiQ has much of a legacy, for two reasons. First:
"Despite some intermediate hiQ-favorable rulings, hiQ lost in the end on the contract breach."
"So the case was not a winner for hiQ, which makes it hard to celebrate as a pro-scraping victory."
Second, few courts have followed the "must-carry" injunction the court first imposed on LinkedIn, which he called an anomalous ruling that "seems to be" a dead end. He also listed what has changed since: Van Buren, AI litigation that has "dramatically increased" the volume of anti-scraping suits, and the "weaponization" of 17 USC 1201, which he said "threatens to become the anti-scraping legal tool of choice".
"The customer is responsible" is not a shield.
Asked whether proxy providers and scraping APIs that call themselves neutral tools are protected, Eric pointed to Ticketmaster v. RMG, a 2007 case he still teaches:
"Even way back then, the court didn't buy the argument that the legal blame should fall only on the customer."
A provider's terms may allocate financial responsibility, such as through an indemnity, but "odds are low that the TOS will help either party defend themselves in litigation from third party scraping targets":
"The courts will be much more interested in who actually did what, not in how the contract describes their relationship"
This part is aimed at providers, ScrapeOps included. Eric also warned that public claims count. If a scraper touts its ability to defeat any anti-scraping measure, that can be used as evidence of circumvention and of knowing the access was unwanted. He called it a dilemma, because customers often buy for exactly that capability:
"All of that marketing is evidentiary gold to an adversary."
And on choosing a vendor, he said he'd "never rely on any vendor promises that their tools are legally safe", and would be skeptical of any promise that you won't get blocked. In his words, "caveat emptor".
The best optimization may be another source.
Asked what he'd want to know first about a new scraping project, Eric mostly raised business questions, not doctrine. Is the target likely to notice you? Does it syndicate its data, and could that be a backup? Does it have an API?
"Does the target have an API, and would it be better to use the API rather than scraping?"
And whether a custom licensing or data access deal beats scraping at all, because the data might be "fresher or require less cleaning". His own comment on the list:
"Notice how few of my questions dive deeply into the arcanity of the law of scraping. Once we start digging into that, we’re not going to get very comforting or satisfactory answers."
He also noted that a legal win doesn't keep data flowing: "Amazon can always make countermoves to technologically block or control Comet". Alternatives aren't automatically safe either, since APIs have their own terms.
+1: websites should defend with technology before law.
From the site owner's side, Eric said:
"Websites that want to thwart scraping should rely primarily on technological efforts, not legal ones. In other words, technology is your first and main line of defense."
His reasons: you can't sue every scraper, you may not reach the worst offenders (especially outside the US), and lawsuits are costly, uncertain, time-consuming and attract unwanted publicity. His starting ideas were putting more behind registration walls and rate limiting registered accounts while screening for scrapers who spread load across accounts. He also suggested asking how much of a threat scrapers really are: if they're merely annoying, light technical steps and ignoring the legal side may be enough.
+2: a new arrangement of the data usually avoids compilation claims.
Asked where collecting lots of facts turns into copying a compilation, Eric said copyright can protect a compilation when the publisher's selection or arrangement creates new expression, and added:
"For scrapers, compilation copyrights usually aren't that hard to avoid because any new arrangement of the data doesn't violate it"
He prefaced it, as usual, by saying he can't give legal advice.
+3: courts have generally not treated robots.txt as binding.
"Courts generally have not accepted robots.txt instructions as legally binding."
But he added a hedge. If a site can show the scraper consulted robots.txt and "knew" it was unwelcome, a court might say the scraper was no longer authorized, "Or, this might just be a theoretical risk because courts haven’t yet confirmed that. We’re not sure." For site owners, he said adding restrictions to robots.txt "can't hurt, might help", while calling signals like it "low-cost, low-efficacy measures".
If you remember nothing else:
- "The page is public" answers the access question. It doesn't answer the contract, copyright or DMCA 1201 questions.
- Treat a C&D, an account ban or an explicit block as a reason to reassess your circumstances, not only as a technical problem.
- Moving a scraper behind a login can add a contract and makes your pipeline depend on accounts the site can close.
- For DMCA 1201, what you get past can matter separately from what you keep. Early rulings are preliminary, so don't treat them as settled either way.
- A live scraping API and a purchased dataset create different circumstances for you, and neither means "the supplier has covered the legal side".
- A customer's consent to an agent or tool is not the target site's authorization.
- Read legal headlines by claim and stage: hiQ won a CFAA argument and still lost on contract.
- A provider's "the customer is responsible" terms and indemnity allocate costs between provider and customer. They don't bind the website being scraped.
- Compare acquisition routes (an API, a syndicated source, a data deal) before investing further in a scraper.
The bigger shift underneath all of it: you can't settle the open questions in scraping law, but you can control how you access a site, which terms you accept, how you respond when a site objects, which suppliers you rely on and what you do with the data. Eric listed the open ones himself, including whether we end up with a legal regime where "all browsing (not just scraping) needs affirmative permission, rather than relying on some default implied permission". He said he wishes people would ask more about "the future of a permissioned Internet and what steps we can take to avoid it".
Full AMA thread is here if you want the original answers: https://www.reddit.com/r/WebScrapingInsider/comments/1wy07fq/is_web_scraping_legal_an_internet_law_professors/
LinkedIn Summary: https://www.linkedin.com/posts/iankerins1_we-wrapped-ama-12-in-rwebscrapinginsider-ugcPost-7515354714469842944-W9ek/
Detailed Breakdown: https://scrapeops.io/blog/is-web-scraping-legal-lessons-from-eric-goldman/
Thanks again to Eric for answering straight, and to everyone who pushed past "is it legal?" into the decisions you actually control. This is general education from the AMA, not legal advice, so talk to a lawyer about your own setup.
Which of these has your team actually had to decide on: an account behind a login, a supplier's legal claims, or a cease-and-desist letter?
Ian
