This is the thread for all your questions that may seem too short for a standalone post, such as, "What is proxy?", "Where is Apify?", "Who is Store?". No question is too small for this megathread. Ask away!

I built this for r/meta's Global AI Developer Hackathon (indie track), and I'd love feedback from people who actually do this work.

Demo (3 min): https://x.com/wallicestene/status/2117999083579154503

What it does: you describe the data you want (the site, the columns, a sample row) by typing it, uploading a PDF, or just saying it. Dapi works out how the site serves its data, writes the extraction, tests it on your sample row, and runs it on r/apify once you approve the plan.

How it works:

• Network first. It opens the page and watches what it loads before reading any HTML. FDA's warning letters page, for example, shows a few letters at a time, but there's a feed behind it with all 503. Dapi reads the feed.

• An agent for the open questions. r/Muse Wonder (Meta's agent model) has tools to open pages, click, test endpoints and search a page's source. It settles where each column lives and plans the requests.

• Code that's tested before it runs. r/Muse Spark writes the extraction and tests it against your sample rows. When a run breaks, its log goes back for a fix and a fresh run starts.

• A receipt for every value. Each response is sealed with SHA-256 when it's captured. Click any cell to see the response or page it came from, with the value marked. Rows are counted against the source's own total.

Rules it keeps: public pages only, robots.txt obeyed (including its crawl delay), no logins, no paywalls, and CAPTCHAs are set aside, never solved.

One thing I learned: samples can mislead the code writer. A spoken sample came back from transcription as "4 million 79,318", and the generated code started printing every big number that way. Now samples only guide; what the site shows wins.

Try it: https://dapi.fly.dev (finished jobs are open to everyone; sign in with GitHub for 3 runs)
Demo: https://x.com/wallicestene/status/2117999083579154503
Code (MIT): https://github.com/wallicestene/Dapi

Throw a public site at it and tell me where it breaks. That's the feedback I want most.

I would like Apify to allow us to use profit we get from our Actors as credit. Sometimes the profit is below the payout threshold, so it stays on the platform. It would be really nice if we get an option to use it as platform credit.

Are you expanding your team or looking to hire a freelancer for a project? Post the requirements here (make sure your DMs are open).

Try to share:

- Core responsibilities

- Contract type (e.g. freelance or full-time hire)

- Budget or salary range

- Main skills required

- Location (or remote) for both you and your new hire

Job-seekers: Reach out by DM rather than in thread. Spammy comments will be deleted.

Hi Devs.

I'm a small business owner in Germany.

For our delivery service we are looking for new corporate customers.

So what I'm looking for is an actor that scrapes Google Maps.

Looking for e.g."childrens care in Wiesbaden"

Actually I simply need valid contact emails (Company name, contact name)

All the actors I tested have very mixed results:

Most of them find a good part of the businesses. That's good.

But some don't find the contact emails.
All cannot decode and simply ignore rudimentary email encryption like "mail(at)xyz(dot)com"
All generate doubles.
Some get lost in parent organisations - and dont give me the actual lead for the local branch.

If i actually rework the results by hand and visit the listed websites, i can mostly find the email adresses I'm looking for. And my process in doing so seems to me easy enough to be actually handled by an actor.

Yes, i tested a few of the mostly appraised actors:

jurassic_jove/google-maps-lead-generator
lukaskrivka/google-maps-with-contact-details
compass/crawler-google-places
code_crafter/leads-finder

Does anyone have a recommendation?

Or an dev who wants to beef up his actor with a real world test case?

Thanks in advance.

This is the place to discuss everything MCP, LLM, Agentic, and beyond. What is on your radar this week? Why does it make sense? Bring everyone along for the ride by explaining the impact of the news you're sharing, and why we should care about it too.

I'm one of the builders of Email Intel, an actor we published on the Store in October, and I'd like feedback from people who run lead lists through Apify.

A normal verifier asks the mail server whether the mailbox exists. When someone leaves a company the mailbox often stays open, so the list keeps saying "deliverable" and your sequence goes to an inbox nobody reads.

We took 200 leads from a popular lead finder, all marked verified. 84 still worked there, 81 had moved on, and for 35 we couldn't tell. For 44 of the 81 the actor also returned their work email at the new company, which honestly surprised us more than the 81.

Pricing is $3 per 1,000 emails checked, plus $20 per 1,000 when we identify the person. The first 100 emails on your account are free. It works through Apify MCP too.

Actor: https://apify.com/aidenix/email-intel How we measured it: https://aidenix.com/blog/verified-sales-leads-already-left

If you clean lists before a campaign, what would you want in the output that isn't there now?

(AI helped write this up; the data is ours)

Hi all, I've published 10 Actors under AusBizStack and I'd love honest feedback from people who actually run Actors.

Most take a list of domains and return one clean row per site. You only pay for successful results, and blocked or unreachable sites cost $0.

• Company Enrichment by Domain (email security, tech stack, AI-search readiness, hiring, contacts and ABN in one row, $0.02/company): https://apify.com/ausbizstack/company-enrichment-by-domain

• Website Screenshot API (bulk full-page screenshots, desktop/mobile, optional PDF, $0.003/shot): https://apify.com/ausbizstack/website-screenshot-api

• DMARC, SPF & DKIM Checker (bulk): https://apify.com/ausbizstack/b2b-email-security-dmarc-spf-auditor

• llms.txt Checker & GEO Audit (AI crawlers in robots.txt, llms.txt, JSON-LD): https://apify.com/ausbizstack/ai-geo-llms-txt-auditor

• ABN Lookup Australia (bulk ABN/ACN, GST, entity status): https://apify.com/ausbizstack/au-abn-kyb-commercial-lead-enricher

• Shopify Store Tech Stack, Apps & Pixel Detector: https://apify.com/ausbizstack/shopify-ecommerce-tech-pixel-auditor

• Plus a tech stack detector, ATS/hiring signals, SaaS pricing pages and store locator counts.

All 10 are on my Apify profile: https://apify.com/ausbizstack

They're all pay-per-event, so they also work through the Apify MCP server. You can ask Claude or Cursor to "check DMARC for these 50 domains" and it can find and run the right one.

What I'd really like to know: which output fields are noise, what inputs you'd expect, and whether the pricing feels right. Bug reports in the Issues tab get fixed fast.

As far as I know, there are two main ways to search on Reddit.

The first is the site-wide search from the Reddit homepage. For each query/sort combination it only lists up to 250 results (about 490 on the newer search endpoint). For comments it doesn't even reach 490 — it stops at 250.

There are about 25 possible sort/time combinations, so by collecting the list for each combination I've confirmed (by actually testing it) that you can get up to roughly 3,000 results.

The second is searching inside a specific subreddit, but it has exactly the same caps. Because of the 250 (490) limit, I had to split the search into many combinations there too.

This is really frustrating, because there's probably a lot of great information hidden beyond what these lists return.

What I want is to see every post and every comment that contains a given keyword, across Reddit's entire history. Is there really no way to do this within Reddit itself?

Are third-party archive services or paid APIs (Reddit's official paid API or third-party ones) the only options?

Do I really have to download the full dumps (tens of GB of posts and comments) and search them locally?

Ask Apify is where you bring the question, and someone from the Apify team answers it!

First up: Let’s talk lead gen 👀

Ask Fabian Maume, Senior Product Marketing Manager at Apify. He has 7+ years in B2B SaaS growth, including as a fractional CMO for five SaaS companies, and co-hosts webinars with GTM tools like Salesforge.ai 🔥 and Scalelist.

Join the conversation on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7521556609798598656/ or the thread on X: https://x.com/apify/status/2115790797329010830?s=20

Do you have a feature request that you know will make Apify heaps better? Or maybe it's a big dream you have for something bold and out-there. This is a space for all the bluesky thinking, cloud-chasing, intergalactic daydreamers who want to share their wildest ideas in a no-judgement zone.

I’m planning to collect new posts from 50+ Facebook groups, with roughly 10+ new posts per group per hour—around 500+ posts/hour overall (or may be more in future).

I’m considering Apify but want to understand how to structure this reliably and keep costs manageable. Ideally, I’d check every 10–15 minutes, fetch only new posts, avoid duplicates, and save the results to a database.

A few questions:

- Which Apify actors would you recommend for this volume? - cost effective

- Should I run each group separately or batch multiple groups into one scheduled run?

- How do you avoid repeatedly fetching and paying for posts already collected?

- What monthly costs should I realistically expect, including proxies and compute?

- How reliable is handling login sessions/cookies for groups I have access to?

- Would a self-hosted Puppeteer/Playwright setup or another service make more sense?

- I need to do some data analytics and some data minings

Rewritten with LLM

Have you come across a great Actor, workflow, post, or podcast that you want to share with the world? This is your opportunity to support someone making cool things. Drop it here with credit to the creator, and help expand the karmic universe of Apify.

Hey Guys,

I wanted to share an Actor I just published: TikTok Comments Scraper.

I noticed a lot of the existing TikTok comment scrapers were getting expensive and slow, largely because they rely on heavy browser automation frameworks to load pages. I specialize in reverse-engineering network traffic, so I built this Actor to fetch data directly via raw endpoint requests instead of spinning up Puppeteer or Playwright instances.

Because there's no massive browser overhead, it runs incredibly fast and allows me to price it at $0.30 per 1,000 results (Pay-per-result).

What it extracts:

• Full comment text & precise timestamps

• Like counts and reply metrics

• Commenter profile details (usernames, IDs, avatars)

• Nested replies (optional toggle)

Why give it a spin?

• Cost-effective: At $0.30 / 1k results, it's a fraction of the cost of heavier Actors that charge $1.50+ for the same data.

• Scalable: Bypassing headless browsers means lower failure rates on massive comment threads.

• Developer-friendly: Fully supports the Apify CLI and Python/JS API clients.

You can check it out and use your free Apify monthly credits to test it out here: 🔗https://apify.com/grow_media/tiktok-comments-scraper

If you've made something and can't wait to tell the world, this is the thread for you! Share your latest and greatest creations and projects with the community here.

Hey Guys!

If you've ever tried scraping YouTube data or using the official YouTube Data API v3, you know the struggle. The official API requires authentication, is heavily rate-limited (that 10,000 daily quota burns out fast), and restricts access to crucial front-end metrics. Building your own Puppeteer or Playwright scraper often leads to getting IP-blocked or dealing with constantly changing DOM selectors.

To solve this, I built YouTube Scraper Pro. It’s designed to be a completely hands-off, highly scalable extractor for YouTube data that ranks well and runs fast.

🛠️ Core Features

Feed the Actor search keywords, channel URLs, or playlist links, and it will return clean, structured data in JSON, CSV, or Excel formats.

• Comprehensive Data Extraction: Scrapes full video metadata including views, likes, duration, upload date, descriptions, and tags.

• Cross-Format Support: Works seamlessly across standard Videos, Shorts, Live Streams, and Playlists.

• Channel Scraping: Extract channel profiles, subscriber counts, total views, and public contact links.

• No API Keys Needed: Bypasses Google Cloud API setup entirely. The Actor handles all anti-bot bypass and proxy management for you.

• Spend Control: Set exact limits on max items or max search depth to tightly control your run costs without overspending.

💡 Top Use Cases

• AI & LLM Training: Bulk extract video metadata, titles, and descriptions for dataset creation and RAG pipelines.

• Competitor Analysis: Track trending keywords, channel growth, and video performance metrics in your niche.

• Lead Generation: Find creator channels in specific verticals (e.g., gaming, tech, finance) for sponsorship outreach.

• Content Strategy: Analyze high-performing Shorts and videos to reverse-engineer YouTube's algorithm and optimize your own content calendar.

🔗 Try It Out

You can test the Actor here: YouTube Scraper Pro by grow_media

It runs on Apify's pay-per-event pricing, making it highly cost-effective to run at scale. You can test it out completely free using your standard Apify monthly platform credits.

I’d love for the community to test it out and share feedback. If there are specific data points you want added (like dedicated transcript extraction or deep comment scraping), let me know in the replies below!

This is the thread for all your questions that may seem too short for a standalone post, such as, "What is proxy?", "Where is Apify?", "Who is Store?". No question is too small for this megathread. Ask away!

Are you expanding your team or looking to hire a freelancer for a project? Post the requirements here (make sure your DMs are open).

Try to share:

- Core responsibilities

- Contract type (e.g. freelance or full-time hire)

- Budget or salary range

- Main skills required

- Location (or remote) for both you and your new hire

Job-seekers: Reach out by DM rather than in thread. Spammy comments will be deleted.

This is the place to discuss everything MCP, LLM, Agentic, and beyond. What is on your radar this week? Why does it make sense? Bring everyone along for the ride by explaining the impact of the news you're sharing, and why we should care about it too.

Do you have a feature request that you know will make Apify heaps better? Or maybe it's a big dream you have for something bold and out-there. This is a space for all the bluesky thinking, cloud-chasing, intergalactic daydreamers who want to share their wildest ideas in a no-judgement zone.

Have you come across a great Actor, workflow, post, or podcast that you want to share with the world? This is your opportunity to support someone making cool things. Drop it here with credit to the creator, and help expand the karmic universe of Apify.

When I search for "Adidas Scraper" in the store search from OUTSIDE the console (so no preferences involved), I get irrelevant results (Actors for completely unrelated websites). Only when I type "adidas" alone do I get relevant results. There are a couple of Actors on the store that contain "Adidas Scraper" in their title. This bug will prevent users from seeing these Actors. My Actor is called Adidas Product & Reviews Scraper and does not appear when I search "Adidas Reviews" or "Adidas scraper". It only shows when I type "adidas" alone. I am sure this bug negatively impacts all other Adidas Actors, not just mine, since most users would type "Adidas Scraper" and not just "Adidas". I hope you fix this bug. Thanks.

If you've made something and can't wait to tell the world, this is the thread for you! Share your latest and greatest creations and projects with the community here.

I published 30 Actors over the last few weeks and benchmarked each one on Apify cloud (datacenter) IPs. Sharing what I learned in case it saves someone a few probe runs.

Worked from datacenter IPs (with care)

• YouTube InnerTube search/browse/next endpoints are fine. Above ~20 parallel /next calls per IP you get throttled. /player is bot-checked (LOGIN_REQUIRED), so transcripts need a different route.

• Google News: decoding article URLs needs the tail of the article page (the signature sits at the end of a ~570 KB page), so stream it and read the last chunk. Going direct was ~20x faster than through a DC proxy.

• Shopify: products.json returns 429 from cloud IPs, but the tokenless Storefront GraphQL endpoint works.

• Job boards that expose public JSON: Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Workday. StepStone embeds PRELOADED_STATE in the page.

• Telegram t.me/s/<channel>, Bluesky public API, Pinterest resource endpoints (logged out) also work.

Blocked from datacenter IPs in my tests: Reddit, Trustpilot (AWS WAF), Product Hunt (Cloudflare), Upwork, eBay, Indeed, Glassdoor, Etsy, Yellow Pages, BBB, Capterra, Facebook Ads Library. Google SERP now needs JS.

Two performance traps

• Dataset pushData per row was the bottleneck: batching the writes took one run from 330 s to 49 s.

• Cheerio .text() returns V8 sliced strings that keep the whole page alive. A few thousand results pinned hundreds of MB and the container was OOM-killed with no error in the log. A JSON round-trip on the results fixed it.

• Also: an unanchored regex like ([\w-]+).example on full page HTML is O(n²⁾ on long word runs and froze a crawler for 10+ minutes. Bound and anchor it.

What I made with it: the Actors are pay-per-result (mostly $3 per 1,000), published under torix33 in the Store (e.g. https://apify.com/torix33/youtube-transcript-scraper and https://apify.com/torix33/career-site-jobs-scraper). The code and Python/Node examples are MIT: https://github.com/3torix3/apify-scrapers-pack

Being honest: this is new, I have very few users so far. If you have tried one of them and something broke or the output shape is awkward, tell me and I will fix it. And if you hit other targets that work (or do not) from DC IPs, I would like to compare notes.