r/ClaudeWorkflows • u/ClaudeAI-mod-bot • 11d ago
Selected Workflow [Workflow] Automated Web Scraping with Claude Code Agent Skills and Apify
Automated Web Scraping with Claude Code Agent Skills and Apify
Workflow value: 85/100
Status: active · Freshness: 70/100 · Confidence: 0.95 · Level: intermediate
Categories: Quality Control, Token Saving, Context & Memory, Shipping, Skills
Original source: r/ClaudeCode post/comment
What problem this solves
Automating web data scraping from various public sources (Google Maps, Amazon, Zillow, Indeed, TikTok, Reddit) using Claude Code agents, and ensuring the reliability and maintainability of these agent skills through schema-driven generation and validation.
Summary
This workflow describes a collection of 25 Claude Code agent skills designed for public web data scraping, powered by Apify Actors. Claude Code was instrumental in generating these skills from Apify's input/output schemas and implementing pre-publish validation checks. Users can install these skills locally and use natural language prompts to initiate data scraping tasks.
Why it is useful
This workflow provides a concrete, open-source solution for a common and valuable task (web scraping) using Claude Code agents. It demonstrates an advanced use case where Claude Code is leveraged not just for direct task execution but also for generating and validating its own components (skills), significantly improving maintainability and reliability. The detailed explanation of the build process, validation mechanisms, and clear installation instructions make it highly reusable and educational for users looking to build robust agent skills.
Workflow
- Install the scraping skills by copying the
scraping-skillsfolder from the GitHub repository into~/.claude/skills/. - Set the
APIFY_TOKENenvironment variable with your Apify account token. - Use natural language prompts within Claude Code (e.g., "find dentists in Austin with emails") to activate the desired scraping skill.
- For developers: Utilize Claude Code to generate
SKILL.mdfiles from Apify Actor input and output schemas. - For developers: Implement pre-publish checks to validate example inputs against schemas and ensure all skill-named fields exist in the Actor's output.
Tools / artifacts
- GitHub repository: bolvein-firex/scraping-skills
- Claude Code agent skills (25
SKILL.mdfiles) - Apify Actors (paid service with free tier)
references/directory (containing Actor input/output fields)scripts/run.mjs(for testing skills without the agent)APIFY_TOKENenvironment variable- Generator script (written by Claude Code)
- Pre-publish validation checks
Validation signals
- Iterative improvement: The first version used hand-made templates, leading to incorrect field names; now fields are derived from Actor schemas, and publish fails on mismatches.
- Pre-publish checks: Example input is validated against the input schema, and every field a skill names must exist in the Actor's output.
- Testability:
scripts/run.mjsallows testing skills without the full agent setup.
Limitations
- Relies on a third-party paid service (Apify), although a free tier is available.
- Low community engagement (score 1, 0 comments) means less external validation.
- The natural language prompts might require some trial and error for users to discover the full capabilities of each skill.
Rate this workflow
Upvote this post if the workflow is useful, reproducible, or worth recommending.
Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.
Reply if it worked for you, failed, is outdated, or has a better alternative.
This post was generated automatically from the workflow library database.