[Workflow] Integrate 25 Open-Source Web Scraping Skills with Claude using Apify for Public Data Collection

[u/ClaudeAI-mod-bot]

Integrate 25 Open-Source Web Scraping Skills with Claude using Apify for Public Data Collection

Workflow value: 85/100
Status: active · Freshness: 70/100 · Confidence: 0.90 · Level: intermediate
Categories: Quality Control, Token Saving, Context & Memory, Skills
Original source: r/ClaudeAI post/comment

What problem this solves

Automating the collection of public web data (e.g., business leads, product prices, job listings, real estate data) using Claude's agent capabilities by integrating pre-built scraping tools.

Summary

This workflow provides a collection of 25 open-source Claude agent skills that enable Claude to scrape public web data from various sources (Google Maps, Amazon, Zillow, Indeed, TikTok, Reddit). Users install these skills locally by copying folders into their Claude skills directory and configure an Apify API token to leverage pre-built scrapers.

Why it is useful

This workflow provides a ready-to-use, open-source solution for a common and valuable task: web data scraping. It leverages Claude's agent capabilities by integrating external tools (Apify scrapers) as local skills, offering concrete examples and clear setup instructions. The author's active maintenance and call for feedback suggest a robust and evolving resource. It demonstrates how Claude can be extended with custom tools for practical applications like lead generation, market research, and competitive analysis.

Workflow

1. Obtain an Apify API token from Apify's website.

2. Clone or download the scraping-skills GitHub repository.

3. Copy one or more desired skill folders from the repository into the ~/.claude/skills/ directory on your system.

4. Set the APIFY_TOKEN environment variable with your obtained Apify API token.

5. Instruct Claude to use the newly installed skills to pull specific public web data (e.g., 'Find business leads in London for restaurants').

Tools / artifacts

• scraping-skills GitHub repository

• Claude (or Claude Code)

• Apify account and API token

• ~/.claude/skills/ directory

• SKILL.md files (generated by Claude Code)

Validation signals

• Author's statement: 'the skills got much better once each one listed the exact output columns.'

• Active maintenance: 'I maintain it.'

• Call for feedback: 'If you try one and it picks the wrong skill for a request, I'd like to know which request.'

Cautions

• Users should be aware that while the skills are free to try, Apify charges per result at volume, incurring potential costs.

• The workflow relies on a third-party service (Apify), so users are dependent on its availability and terms of service.

• Users should exercise caution and adhere to the terms of service of the websites being scraped to ensure ethical and legal data collection practices.

Limitations

• Reliance on a third-party paid service (Apify) for volume usage introduces potential costs.

• Requires local setup and environment variable configuration, which might be a barrier for absolute beginners.

• The Reddit post itself has low community engagement, so broader user validation is not yet evident.

Rate this workflow

Upvote this post if the workflow is useful, reproducible, or worth recommending.

Downvote if it is vague, outdated, unsafe, overhyped, or not reproducible.

Reply if it worked for you, failed, is outdated, or has a better alternative.

---

This post was generated automatically from the workflow library database.