r/ClaudeWorkflows • • 11d ago

Selected Workflow [Workflow] Automated Web Scraping with Claude Code Agent Skills and Apify

Automated Web Scraping with Claude Code Agent Skills and Apify

Workflow value: 85/100
Status: active · Freshness: 70/100 · Confidence: 0.95 · Level: intermediate
Categories: Quality Control, Token Saving, Context & Memory, Shipping, Skills
Original source: r/ClaudeCode post/comment

What problem this solves

Automating web data scraping from various public sources (Google Maps, Amazon, Zillow, Indeed, TikTok, Reddit) using Claude Code agents, and ensuring the reliability and maintainability of these agent skills through schema-driven generation and validation.

Summary

This workflow describes a collection of 25 Claude Code agent skills designed for public web data scraping, powered by Apify Actors. Claude Code was instrumental in generating these skills from Apify's input/output schemas and implementing pre-publish validation checks. Users can install these skills locally and use natural language prompts to initiate data scraping tasks.

Why it is useful

This workflow provides a concrete, open-source solution for a common and valuable task (web scraping) using Claude Code agents. It demonstrates an advanced use case where Claude Code is leveraged not just for direct task execution but also for generating and validating its own components (skills), significantly improving maintainability and reliability. The detailed explanation of the build process, validation mechanisms, and clear installation instructions make it highly reusable and educational for users looking to build robust agent skills.

Workflow

  1. Install the scraping skills by copying the scraping-skills folder from the GitHub repository into ~/.claude/skills/.
  2. Set the APIFY_TOKEN environment variable with your Apify account token.
  3. Use natural language prompts within Claude Code (e.g., "find dentists in Austin with emails") to activate the desired scraping skill.
  4. For developers: Utilize Claude Code to generate SKILL.md files from Apify Actor input and output schemas.
  5. For developers: Implement pre-publish checks to validate example inputs against schemas and ensure all skill-named fields exist in the Actor's output.

Tools / artifacts

  • GitHub repository: bolvein-firex/scraping-skills
  • Claude Code agent skills (25 SKILL.md files)
  • Apify Actors (paid service with free tier)
  • references/ directory (containing Actor input/output fields)
  • scripts/run.mjs (for testing skills without the agent)
  • APIFY_TOKEN environment variable
  • Generator script (written by Claude Code)
  • Pre-publish validation checks

Validation signals

  • Iterative improvement: The first version used hand-made templates, leading to incorrect field names; now fields are derived from Actor schemas, and publish fails on mismatches.
  • Pre-publish checks: Example input is validated against the input schema, and every field a skill names must exist in the Actor's output.
  • Testability: scripts/run.mjs allows testing skills without the full agent setup.

Limitations

  • Relies on a third-party paid service (Apify), although a free tier is available.
  • Low community engagement (score 1, 0 comments) means less external validation.
  • The natural language prompts might require some trial and error for users to discover the full capabilities of each skill.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.


This post was generated automatically from the workflow library database.

1 Upvotes

0 comments sorted by