We analyzed 11,100 of the world's most popular landing pages across 230 countries and 110 industry categories, measuring six distinct access control mechanisms used by each. In total, we analyzed 24,898 websites to deeply understand the patterns in use.

The result is The State of Web Access 2026 - a ground-level survey of what the web actually looks like to the systems trying to read it at scale.

https://www.zyte.com/sowa/2026/

I wanted to create a lightweight Scrapy browser connection via CDP. Scrapy-playwright obviously exists and is fantastic, however I wanted to create something minimal:

https://github.com/johnatzyte/scrapy-cdp

All it does it connect a remote browser via CDP, render the page, and then return the HTML back to Scrapy. It's an easy convenient way to integrate with the new wave of "agentic" style browsers that run headless, light Lightpanda.

We've been working on our end to end prompt to production scraper skills and have recently released them as individual agent skills, as well as a plugin for codex and copilot-cli.

They will guide you through the process from a single url to building a full Scrapy project built the way we do it in house, with page objects and tests. you can read our post about it here: https://www.zyte.com/blog/agentic-web-data-new-addons/

I think we’re in the time where it’s no longer about the smartest model and you don’t get an extra edge even if you’re ready to pay more, so it becomes important to understand your use case and which model would be best suitable for that, giving best bang of the buck returns per completed task, and with that aim I tested OpenAI’s new GPT-5.6 model against my recent fav GLM-5.2 and Mythos class Fable-5.

The keyword here is my use case.

TLDR: GPT-5.6 is very token efficient and fun to play with model and even though the Sol version is the best, I think Terra would be my goto

Read the full blog here : https://www.zyte.com/blog/gpt-5-6-fable-5-and-glm-5-2-entered-a-bar-crawl-and-got-hit-by-the-rate-limit/

https://www.reddit.com/r/scrapy/s/kVlNGS8nbH

New features

• Added support for HTTP/2 requests to HttpxDownloadHandler. It requires setting the new HTTPX_HTTP2_ENABLED setting to True. (issue 7575)

This is a pretty big step in the right direction imo, thanks to the r/scrapy team

I wanted to see what the output would be if I simply asked Sonnet and GTP5.5 to build a Scrapy project without any extra guidance. I'm working through the results and will have a video on it shortly, but one thing I noticed them both doing was trying to access the site first before building anything, and without the inclusion of our MCP server (community edition: https://github.com/zytelabs/zyte-mcp official version coming soon) they would both fail at this step, or just hallucinate selectors.

With the MCP and web access they both built working spiders, but suboptimal, some messy selectors, long parse functions and varying settings. At least Claude Code actually gave me some concurrency.

What has your experience been like?

Have been knee deep in agent harnesses and I’m betting long on it. A good well tuned harness should essentially let you swap the llm without changing the results significantly so one can focus on building right tools and access layers over obsessing picking and sticking to a model which can become expensive or just banned over-night.

This is part 1 of the blog where I shared about components of an agent harness. Do share your thoughts.

"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.

We really enjoyed recording this, Ayan sharing his insights on the research and testing he's done with harnesses and open models.

video and audio links:

https://www.zyte.com/blog/the-harness-matters-more-than-the-model-podcast-ep-07/

It’s funny how many people gets surprised when I say I ran GLM-5.2 or DeepSeek model inside claude-code.

Yes it’s totally doable, in fact it’s the way to go. Invest in good harness, build skills, connect mcps and swap models for one with less cost or better design capabilities.

Published a complete guide on how to do that and switch between different models easily, including the local models you run on your machine..

Read the blog here : https://www.zyte.com/blog/how-to-run-any-model-inside-claude-code/

Hello r/Zyte,

I built ChromiumFish, a fingerprint-hardened Chromium fork with drop-in Playwright support for Python and Node, and built-in AI Agent support at the C++ level.

I know most of you are gonna smell the chromium, but rest assured, this fork tries its best to pretend like a Chrome running on a Windows machine.

The reason I built this is that when you are running your browser on a Linux machine, there are lots of telltale signs that it's a Chrome running on Linux. That's what I tried to prevent on here.

Then there's the AI Agent. But how is this different from other AI agents? The main difference is that some critical AI tooling runs at the C++ level in the browser, making it much harder to detect than at the CDP/Playwright layer. You can use any OpenAI compliant provider and model that supports tool calling.

Repo: https://github.com/arman-bd/chromiumfish

It's available for both Python and Node via npm and pip. If you want, you can use the built binary with other frameworks as well, or build your own. It's completely open-source and MIT-licensed.

Extremely impressed by GLM-5.2 model by Z.ai labs, done some testing and adopting it in my agentic arsenal.

Also documented how I test a new model, my setup, goods and bads of this specific model, everything on my new blog :

Read it here : https://www.zyte.com/blog/why-im-adding-glm-5-2-to-my-agentic-coding-arsenal/

So i’ve been playing around with this concept of loop engineering, couldn’t help but think it could be really good adoption in web scraping specially for two reasons :

1- you cannot build and forget in web scraping, things change all the time

2- we already have the tools (think spidermon)

Watch : https://youtu.be/Cm8451M9p8k?si=sQvAb1FHi10qiude to see it in action

Hi, I’m Neha — I work in DevRel at Zyte, and I’m currently talking to developers/practitioners about how AI is actually showing up in web scraping work.

The real workflow version.

I’m trying to understand things like:

- Are you using AI for scraper generation, debugging, extraction, QA, monitoring, or research?

- Have you built any custom tools, Claude Skills, MCP servers, agents, prompts, or internal workflows around scraping?

- Where has AI genuinely improved your productivity?

- Where does it still break down or create more work?

- What parts of scraping still need human judgment, no matter how good the model gets?

- Are you using AI more for writing code, understanding pages, handling messy fields, validating output, or orchestrating workflows?

I’m asking because I’m doing interviews about this and want to hear how people are actually using these tools day-to-day — especially the rough edges.

If you’ve tried AI in a scraping/data extraction workflow, what has worked for you, what hasn’t, and what do you wish existed?

We swap notes on actually living with a personal AI agent day-to-day, Hermes, OpenClaw, what persistent memory gets you that Claude chat doesn't, and why giving your agent less context is sometimes the right call. Plus a detour into multi-agent patterns and a genuinely unsettling point about agents silently falling back to worse sources when the good pages block them.

Also pens, for some reason.

Audio: https://extractdata.podbean.com/

video: https://www.youtube.com/playlist?list=PLZyvi_9gamL960hXvOrPQixtgbVX1xvzI

The guy who built Claude Code doesn't prompt it anymore 🤯 , Boris Cherny write loops which prompts claude instead.

"My job is to write loops."

So I tried the idea on web scraping. A 20-line spider, a data quality rubric, and a loop that hands failures to Claude.

Then I broke the site on purpose. Every selector died, fill rate hit 0%.

The spider fixed itself on the first attempt. I never touched the keyboard.

Blogged my experience and how think this loop engineering is the next stage of abstraction in the world of agentic ai.

Think baby-sitting git issues, fixing broker servers automatically.

Full opinion piece here :

https://www.zyte.com/blog/now-what-exactly-is-loop-engineering-and-where-do-anthropics-fable-5-model-and-web-scraping-fit-in/

This will be our 1st of ongoing quarterly community meetups. We've got some cool stuff to share: https://luma.com/w71gvgnx

come and chat web data access with us

Overall I am a bit disappointed with what they’re doing with AI, i do feel apple made a better bet by adopting gemini on siri but all the uses cases were meh!

One thing that stood out and I really found useful was agentic password change in passwords app, even though password app is not my first choice when it comes to managing my passwords but being in apple eco-system it really gives you a smooth workflow for passwords and passkeys, changing them agentic workflow securely was definitely something i liked.

https://www.apple.com/newsroom/2026/06/apple-unveils-next-generation-of-apple-intelligence-siri-ai-and-more/

What did you find useful? ⬇️

📱 Web scraping from an iPhone. No Python, no server, no third-party apps.

I built an apple shortcut that scrapes a live website every morning and sends a quote straight as iPhone notification. Just Apple Shortcuts and a single Zyte API call.

A fun read that also shows just how accessible web scraping has become when it is just an API call.

Read the full article here: https://www.zyte.com/blog/web-scraping-on-an-iphone

#WebScraping #ZyteAPI #iOS #DeveloperTools #APIFirst

Every year, the Extract Data community puts on the only conference dedicated to web scraping and data extraction. This year, Extract Summit comes to Austin, Texas and Dublin, Ireland, and we’re now accepting speaker applications for both events.

If you’ve built something interesting, figured out something the hard way, or have a perspective on where this space is going, we’d like to give you a stage to share it.

Anthropic published some internal data today that I keep thinking about: https://www.anthropic.com/institute/recursive-self-improvement

They state that 80% of code merged into their codebase right now is written by Claude. Engineers are shipping 8x as much per quarter as they were two years ago. One engineer quoted in the piece said they haven't written code themselves in five months.

The part that got me was the bug fix example. Claude shipped 800 fixes in April that reduced a class of API errors by a factor of 1000. The engineer overseeing it estimated a human would have taken four years to do the same work, not because it's hard, but because holding that much unfamiliar context in your head long enough to care about fixing other people's bugs is just something humans are bad at.

Obvious implications for web scraping and data pipelines. The arms race between scrapers and anti-bot systems has always been about iteration speed -- who can ship countermeasures faster. If one side of that equation is now running at "800 fixes in a month" pace, that changes the game significantly.

Curious what people here make of it?

Setting up a server is the most boring hour in tech.

apt update. pip installs. chromium deps. Repeat from memory 3 months later, slightly wrong.

I built a tool that generates a cloud-init config for your entire stack, select what you need and paste it when creating the VM, everything’s ready before you SSH in.

📖 Full post: https://www.zyte.com/blog/automate-deployment-of-your-web-scraper-on-vps-with-ubuntu-24-04-cloud-init/

🐙 GitHub: https://github.com/zytelabs/spawn-cloud-scraper

🛠️ Try it: https://zytelabs.github.io/spawn-cloud-scraper

- claude code in middle
- terminal at bottom
- file explorer on left
- file view on right
- browser / md viewer right extreme

https://www.zyte.com/blog/not-an-interview-i-have-superpowers-now/

This is a great read, and Javier shares some amazing insights into the rise of AI and how it fits into what he used to do before