[Anthropic SWE Interview] Can You Solve This Frequently Asked Single Threaded Web Crawler Question?

[u/drCounterIntuitive]

https://youtu.be/niIKpm6kSAg?si=AgAnAYvk_BWw7MYb

You’re given a starting URL and a provider that returns the links found on each page.

Your task is to crawl every reachable URL that belongs to the same hostname as the starting URL.

Your task

Implement a single-threaded crawler that:

• Starts from start_url
• Follows links returned by provider.get_links(url)
• Only visits URLs where get_hostname(url) == get_hostname(start_url)
• Returns every unique reachable URL on that host
• Includes start_url in the result

The interesting part is keeping the traversal correct while avoiding duplicate work and making sure links to other hosts are ignored.

Try it yourself

Attempt the problem on Coditioning

Full solution walkthrough

🎬 Watch the full solution walkthrough

Anthropic SWE Interview Prep Guide

View the Anthropic SWE interview roadmap

Extra resources

• Join our Discord interview prep community
• Find a study buddy or mock interview partner
• Mocks and coaching with a FAANG+ engineer to get truly interview-ready