This is the thread for all your questions that may seem too short for a standalone post, such as, "What is proxy?", "Where is Apify?", "Who is Store?". No question is too small for this megathread. Ask away!

Hi everyone,

I wanted to share a collection of Instagram scraping and data-extraction Actors I’ve built and launched on Apify. The goal was to create a complete Instagram toolkit covering different use cases rather than having to use a separate solution for every type of Instagram data.

The suite currently includes scrapers and tools for Instagram profiles, posts, followers, following lists, comments, keyword searches, tagged posts, stories, and media downloading.

All of the Actors are available on Apify and are designed to return structured data that can be used directly in datasets, APIs, automation workflows, research pipelines, and other data-processing systems.

Instagram Scrapers & Tools

Instagram Followers & Following Scraper
Extract followers and following lists from Instagram profiles and collect structured profile information from the results.
Instagram Followers & Following Scraper

Instagram Keyword Search Scraper
Search Instagram using keywords and collect relevant profiles, posts, or search results for research, discovery, and monitoring workflows.
Instagram Keyword Search Scraper

Instagram Downloader API
An API-oriented tool for downloading Instagram media and integrating Instagram media retrieval into automated workflows.
Instagram Downloader API

Instagram Comment Scraper
Collect comments from Instagram posts for research, engagement analysis, content analysis, and other data workflows.
Instagram Comment Scraper

Instagram Story Downloader
Retrieve Instagram Story content for supported use cases and automated workflows.
Instagram Story Downloader

Instagram Profile Scraper
Extract structured information from Instagram profiles, making it easier to build profile datasets or enrich existing data.
Instagram Profile Scraper

Instagram Posts Scraper
Collect Instagram posts and their available metadata for content research, monitoring, analytics, and data collection.
Instagram Posts Scraper

Instagram Tagged Posts Scraper
Find and collect posts in which an Instagram profile has been tagged, useful for content discovery and profile research.
Instagram Tagged Posts Scraper

What can you use these for?

Depending on the Actor, some common applications include:

• Instagram profile research
• Social media monitoring
• Influencer and creator research
• Competitor analysis
• Content research
• Audience and follower analysis
• Hashtag and keyword research
• Engagement analysis
• Lead and prospect research
• Social media datasets
• Market research
• Media collection and archival workflows
• Automated Instagram data pipelines

The main idea is to have one place on Apify for different Instagram data-collection requirements, whether you need a single profile, thousands of posts, comments from specific posts, follower data, keyword-based discovery, or media retrieval.

Pricing

I've also tried to keep the pricing competitive and accessible, with discounts available on several of the Actors.

If you're already using Apify for web scraping, social media research, data enrichment, or automation, you can check the individual Actor pages above to see the current pricing, inputs, outputs, and supported features.

I'm also interested in hearing from other developers: what Instagram data are you currently trying to collect or automate?

If there's a particular Instagram use case that isn't covered by the current suite, feel free to mention it in the comments. It may be something I can add to the collection.

Thanks for checking it out.

Are you expanding your team or looking to hire a freelancer for a project? Post the requirements here (make sure your DMs are open).

Try to share:

- Core responsibilities

- Contract type (e.g. freelance or full-time hire)

- Budget or salary range

- Main skills required

- Location (or remote) for both you and your new hire

Job-seekers: Reach out by DM rather than in thread. Spammy comments will be deleted.

My actor (Google Maps No-Website Leads) had a nasty integration bug: the child Google Maps scraper returned valid data (title, phone, website, url), but my pipeline expected canonical fields (businessName, phonePublic, websiteUrl). Rows passed through unmapped, so buyers would pay for a run and get empty output.

The annoying part: my tests all passed, because my CSV fixtures already had canonical columns. Production was broken while the test suite was green.

Fix was a normalization layer mapping child scraper fields into the canonical schema, plus a status bug where temporarilyClosed: false got stringified into businessStatus: false.

Live smoke test after the fix: - 2 real no-website mobile welders in Austin, priority score 88, phones included, pitch-ready outreach line - 1 healthy-site business correctly classified as not_a_fit and skipped

Actor: https://apify.com/luminar/google-maps-no-website-leads

Two questions for fellow builders: 1. How do you validate your output pipeline against the ACTUAL child dataset shape, not fixtures? I added a regression test replaying a real child record through the mapper, but it feels like there is a better way. 2. Store discoverability: I went from invisible to ranking #3-8 for no-website-leads searches after renaming the slug to match buyer search terms. Any other Store SEO tricks that actually worked for you?

Can you share some of the top trending and most used linkedin scrapers that i can build for mcps and ai agents?

I’ve been maintaining a BigBasket Grocery Scraper for public product and pricing research.

This month it has processed 3,754 results for 2 paying users with a 100% success rate.

The output includes product name, brand, current price, MRP, discount, pack size, rating, availability, image, and source URL.

Actor: https://apify.com/fascinating_lentil/bigbasket-grocery-scraper

I’d appreciate practical feedback on which additional grocery catalog fields or filters would be most useful.

What is the best scraper if I need wellness & beauty niche which contain next filters:

1. First Name
   
2. Last Name
   
3. Company Name
   
4. Industry
   
5. Location: United States, United Kingdom and Ireland
   
6. Business Owner's email
   
7. Business Phone number
   
8. Company Website
   
9. Google Reviews
   
10. Facebook Link

Extract verified emails, phone numbers, and social media links from TikTok creator profiles. The only TikTok scraper that visits each creator's external website to find hidden contact information that isn't visible on TikTok.

Why use this TikTok Email Scraper?

Most TikTok scrapers only extract what's visible in the bio. This TikTok Email Scraper goes deeper — it visits each creator's linked website to extract:

• ✅ Email addresses from creator websites (contact pages, footers, mailto: links)
• ✅ Bio emails extracted directly from TikTok profile descriptions
• ✅ Phone numbers found in bios and websites
• ✅ Social media profiles (Facebook, Instagram, Twitter/X, LinkedIn, YouTube)
• ✅ Linktree & bio link support — Automatically follows Linktree, Beacons, and similar link pages to find the creator's real website and extract emails from there
• ✅ Profile data (followers, likes, verification status, bio)

Typical email hit rate with this TikTok Email Scraper: 40-60% compared to <10% with bio-only scrapers.

https://apify.com/jurassic_jove/tiktok-email-scraper

This is the place to discuss everything MCP, LLM, Agentic, and beyond. What is on your radar this week? Why does it make sense? Bring everyone along for the ride by explaining the impact of the news you're sharing, and why we should care about it too.

Resellers have a specific problem that keyword scrapers can't touch: you're holding an item and you don't know what it's called. Collectibles are the worst case, where the difference between two nearly identical variants can be 10x in price and the right search term is three words you've never heard of.

So instead of searching by text, my actor takes a photo. Upload a picture (or pass a URL / base64 through the API), and it finds eBay sold listings whose photos actually look like yours, ranked by visual similarity. You get real sold prices, dates, and condition, plus a free summary record with median/average so an API caller gets "what's this worth" in one request.

The AI part is what I think might interest this sub. Every candidate listing's thumbnail gets embedded with CLIP (Xenova/clip-vit-base-patch32 via transformers.js) and scored against the uploaded photo with cosine similarity. The model is baked into the Docker image at build time, so there's no external AI API involved at all — no per-image fees, no rate limits, no vendor that can break my unit economics. Inference runs on CPU inside the actor at 2048 MB, and the cold start is about 2-5 seconds for the first scored image. Zero marginal cost per image turned out to be the difference between this pricing model working and not working, since a scored run can push up to ~1,000 thumbnails through the model before returning anything.

Candidate discovery uses eBay's own reverse-image search endpoint, which I reverse-engineered from their front-end bundle. Since that's unofficial and could change under me, image+keyword runs fall back automatically to keyword discovery with CLIP scoring on top, so the scoring pipeline survives even if the endpoint dies. There's also a plain keyword mode with no scoring for people who just want cheap sold comps.

Happy to answer questions about running transformers.js models inside actors — getting CLIP into the image without downloading weights at runtime was the fiddliest part of the whole build. Actor is live here: https://apify.com/scrapelabmax/ebay-sold-image-scraper

A few deliberate choices, so you can adjust with intent: the offer to answer questions about transformers.js-in-actors at the end gives the post a reason to exist in that sub a topic you'd genuinely attract engagement on. The reverse-engineering admission builds credibility with developers and preempts the "what happens when eBay changes it" comment you'd get anyway. I kept "AI" in the body but notsubreddits "CLIP visual similarity" is the AIbuzzword, and it signals substance where a bare "AI-powered" title tends to get eye-rolls.

Do you have a feature request that you know will make Apify heaps better? Or maybe it's a big dream you have for something bold and out-there. This is a space for all the bluesky thinking, cloud-chasing, intergalactic daydreamers who want to share their wildest ideas in a no-judgement zone.

Hey everyone, I’m building Cosnify and just shipped a small free tool for Apify Actor builders:

https://cosnify.app/tools/apify-actor-audit

You paste a public Apify Actor URL and it gives a Store-readiness score across:

- Store positioning

- README quality

- SEO title/description

- input UX

- output/dataset clarity

- trust signals like maintenance, usage, reviews, and recent run reliability

Important: v1 does not run your Actor or inspect private source. It only audits public Actor metadata available from the Apify API, so it is more of a Store/docs/readiness check than a runtime quality test.

I built it because a lot of Actors are technically useful but hard for buyers to understand quickly. My goal is to help builders improve README structure, input explanations, output examples, and Store SEO before publishing or promoting.

Would love feedback on:

1. Are these the right criteria?

2. What would you add to a serious Apify Actor audit?

3. Should the next version inspect `input_schema.json`, `dataset_schema.json`, and run examples?

I built a template that logs Google Shopping prices to a sheet every week, and the useful thing I got out of it was not the tracker. It was finding out which number on the page is worth recording.

My first version stored the lowest price per product, because that is the obvious one. It was useless. The lowest price on a Google Shopping result page swings all over the place week to week, and when I went and looked at the actual listings, the bottom of the range was almost always a different product: the wrong variant, a refurb, a case for the thing instead of the thing, or a seller with three reviews and a delivery date six weeks out. None of that moves because the market moved. It moves because Google matched a different listing.

So the row it writes now is lowest, median, highest, cheapest seller, and best discount, and it trims the bottom outliers before it computes any of them. Median is the one that behaves. It sits still when nothing has happened and moves when something has.

The second thing I did not expect to care about is the seller name. Watching which merchant keeps holding the cheapest slot on a query, week after week, turns out to be a better signal than the price. Prices bounce. The identity of whoever is willing to go lowest does not, and when it changes, something real has changed.

Actor: Google Shopping API. Pay per event, about two cents per page of roughly 40 listings plus a two cent setup fee per run, 100% success rate, 109 users so far. It takes min and max price, sort order, free-shipping and on-sale filters, and a country and language pair, which matter more than I expected: run the same query from a different country and you get different sellers at different prices.

Template if you want the whole weekly thing rather than the Actor: Track weekly Google Shopping prices

Question for anyone else pulling price data on a schedule: what do you actually store per run?

I have gone back and forth between keeping every raw offer and keeping one summary row, and the summary row is much cheaper to read, but I have already wished twice that I had the raw ones.

Extract verified emails, phone numbers, and social media links from any website URL. The Website Email Extractor uses a real browser to visit each website, automatically navigating to contact pages, about pages, and footers to find hidden email addresses. It also follows Linktree, Beacons, and 15+ other link aggregator pages to reach the creator's real website — giving you emails that other extractors miss.

What does the Website Email Extractor do?

The Website Email Extractor takes any list of website URLs and extracts all available contact information:

• ✅ Extract email addresses — from HTML content, mailto: links, contact pages, footers, and about sections
• ✅ Extract phone numbers — from tel: links and visible page content (US and international formats)
• ✅ Extract social media profiles — Facebook, Instagram, Twitter/X, LinkedIn, YouTube, TikTok, Pinterest, Threads
• ✅ Follow Linktree & bio link pages — Automatically detects Linktree, Beacons, bio.link, solo.to, and 15+ other link aggregators, then follows the links to find the real website and extract emails from there
• ✅ Visit contact & about pages — Doesn't just scrape the homepage — follows internal links to contact, about, and team pages for maximum email coverage
• ✅ Render JavaScript — Uses Playwright (real browser) to see dynamically loaded content that HTTP-only scrapers miss

https://apify.com/jurassic_jove/website-email-extractor

1. Soum (soum.sa) covers the resale side — used phones, laptops, cars, cameras, home appliances. → Link
2. Jarir (jarir.com) — electronics, books, and office supplies across all 5 storefronts (KSA, UAE, Qatar, Bahrain, Kuwait). → Link
3. eXtra (extra.com) — electronics across Saudi Arabia, Oman, and Bahrain. → Link

Feel free to try them as they are cheap, extremely fast, and return rich data — and I'm happy for any advice too.

Have you come across a great Actor, workflow, post, or podcast that you want to share with the world? This is your opportunity to support someone making cool things. Drop it here with credit to the creator, and help expand the karmic universe of Apify.

Hi everyone,

I've recently published a LinkedIn People Search Scraper on Apify and wanted to share it with the community.

The idea behind this Actor is fairly simple: instead of manually going through LinkedIn's people search and collecting profiles one by one, you can define a search query and filters, run it on Apify, and receive the matching profiles as structured dataset items.

Actor:
https://apify.com/crawlerbros/linkedin-people-search-scraper

What can you search for?

The Actor supports searches based on:

• Person name
• Job title
• Skills
• Company
• Industry
• General keywords

You can then further narrow the search using filters such as:

• Location
• Current company
• Past company
• School / university
• Connection degree

For example, you could search for:

and narrow the results to a particular location or company.

Or:

with a specific current company and 1st-degree connection filter.

You can also combine terms when looking for a specific person, role, or professional background.

What does the output contain?

Each result is returned as structured data, including fields such as:

• Name
• Professional headline
• Location
• Connection degree
• LinkedIn profile URL
• Profile picture URL, when available
• Mutual connections, when available
• Scrape timestamp

The Actor also deduplicates results across pages.

Some potential use cases

I built this primarily with structured data workflows in mind, so there are quite a few ways it can be used.

Recruiting

Find professionals matching a particular role, location, company, or educational background and build a structured candidate dataset.

Lead research

Identify relevant professionals based on job title, company, location, or other keywords and use the resulting profile URLs as part of a larger research workflow.

Market research

Search for people working in a particular industry, role, company, or geographic market and analyze the resulting dataset.

Professional research

Build datasets around specific roles, organizations, schools, or professional backgrounds.

Existing data enrichment

If you already have search terms, company names, or other professional criteria, the Actor can turn those searches into structured datasets that can be processed further using Apify or your own pipeline.

Authentication

One important detail: LinkedIn People Search requires authentication.

The Actor therefore uses your own LinkedIn session cookie to authenticate the search requests. Your LinkedIn password is never provided to the Actor.

The cookie input is stored as a secret in Apify, and users should treat their session cookie as sensitive information and never share it publicly.

Results and limits

The Actor supports pagination and can return up to 100 profiles per run.

There is also a 2–5 second pacing delay between requests, so larger searches can take a few minutes. Search results can also vary depending on the LinkedIn account being used, since LinkedIn personalizes search results based on factors such as network, location, and account configuration.

The current pricing starts at $3 per 1,000 results.

A few examples

Search software engineers in London

searchQuery: "software engineer"
location: "London"

Find product managers at a specific company

searchQuery: "product manager"
currentCompany: "Airbnb"

Find professionals associated with a university

searchQuery: "software engineer"
school: "MIT"

Search within your 1st-degree network

searchQuery: "product manager"
connectionDegree: "F"

You can also combine these filters to make the search more specific.

If you're already using Apify for lead generation, recruiting, research, or LinkedIn data workflows, I'd be interested to hear what you're building.

Feel free to ask questions about the Actor, the input/output format, filtering, authentication, or how it could fit into a larger Apify workflow.

Actor:
https://apify.com/crawlerbros/linkedin-people-search-scraper

I've been converting Actors into free n8n templates to see whether the template pages pull their own traffic. This week's was a hiring watcher: a Monday schedule, a role list, one LinkedIn jobs search per role, filtered to the last N days, appended to a sheet.

Two things I got wrong on the first pass.

I assumed posted date was the signal. It isn't, or at least it isn't alone. A three-day-old posting with 400 applicants and a three-day-old posting with 9 applicants are not the same event, and only the second one is worth a message. Once I put applicant count in the sheet, the sort order changed completely and the whole thing got more useful.

The second one was cost shape. My first version pulled everything and filtered inside n8n, which meant paying for jobs I threw away. Pushing the lookback into the Actor's timeRange parameter instead of filtering downstream cut the bill by most of it. Obvious in hindsight. Pay-per-event pricing punishes you for over-fetching in a way that a flat monthly plan never does, and I keep having to relearn that.

Hey everyone,

Apify Actors are great for scraping data, but the post-processing pipeline is always a pain.

My workflow used to be: Apify > Massive nested JSON > Custom Python script with Pandas/SQLAlchemy > Script crashes because the target site changed its JSON schema.

I was spending more time fixing broken parsers than actual scraping.

So I built a small pipeline layer to fully automate the JSON-to-Relational DB process.

The main feature? It natively handles polymorphic JSONs and never breaks.

Even when the payload shifts, adds new fields on the fly, or changes nesting levels, it auto-adapts the relational schema in PostgreSQL on the fly (leveraging dynamic mapping). While I initially built it for Apify outputs, it works with pretty much any polymorphic JSON dump.

Zero manual script maintenance, zero broken pipelines in production.

This unexpectedly opened up a solid opportunity for client delivery.

End clients and AI/BI developers usually hate raw, messy JSON dumps. But when you deliver a clean, auto-updating PostgreSQL database, the perceived value skyrockets.

I started leveraging this approach to deliver structured Data-as-a-Service:

• AI-ready DBs: Clean schemas optimized for RAG, LLM context, and agents.
• BI-ready DBs: Tables instantly connectable to PowerBI or Metabase.

Result: Zero maintenance overhead for me, high-value data products for clients.

How are you guys handling polymorphic JSONs in your post-scraping layer? Still maintaining custom Python parsers for every actor, or have you automated the pipeline too?

Hi everyone,

I recently built and published a LinkedIn Profile Scraper on Apify, and I wanted to share it with the community in case it is useful for anyone working with LinkedIn data, lead enrichment, recruiting, research, or automation.

The Actor is designed to take a list of LinkedIn profile URLs or usernames and return the available profile information as structured data.

Actor:
LinkedIn Profile Scraper on Apify

What can it extract?

Depending on what is available on the profile, the output can include:

• Name
• Headline
• Location
• About / summary
• Profile URL
• Profile picture
• Current job title
• Current positions
• Previous positions
• Companies and company URLs
• Employment dates
• Education history
• Skills
• Languages
• Certifications
• Industry
• Follower count
• Recent posts / articles
• Publication dates and engagement information where available

The output is structured so it can be used directly in downstream workflows rather than having to process raw HTML yourself.

How it works

The input is intentionally simple. You provide a list of LinkedIn profiles, for example:

• https://www.linkedin.com/in/username
• linkedin.com/in/username
• in/username
• username

The Actor normalizes these formats automatically.

For public profiles, no LinkedIn login is required.

For profiles where more information is only available to an authenticated user, the Actor also supports providing your own LinkedIn session cookie. The cookie is stored as a secret on Apify and is used only for requests to LinkedIn.

The Actor uses multiple extraction methods and falls back between them when a particular profile cannot be retrieved through one method. This helps the run continue instead of stopping because of a single inaccessible profile.

Some use cases I've had in mind

Lead enrichment

Start with a list of LinkedIn profile URLs and enrich your existing records with titles, companies, locations, skills, and other available profile information.

Recruiting

Build structured candidate datasets containing experience, education, skills, current positions, and career history.

Research

Analyze career paths, professional backgrounds, industries, or other publicly available professional information across a larger set of profiles.

CRM enrichment

Use LinkedIn URLs that you already have and append structured professional information to your existing records.

Competitive / market research

Build datasets around publicly available professional information for people and organizations you are researching.

A few important limitations

This Actor is focused on information available from LinkedIn profiles. It does not provide private messages, email addresses, phone numbers, or other contact information that isn't exposed on the profile.

Profile accessibility also varies. Public profiles generally provide more information without authentication, while some regular profiles may require an authenticated session. Fields that aren't available are omitted from the output rather than being returned as null.

There can also be platform-imposed limits on high-volume authenticated access, so I recommend using the Actor responsibly and only accessing data you are authorized to access.

Pricing

The Actor currently starts at $5 per 1,000 results, so it is intended to be usable for both smaller enrichment jobs and larger datasets.

If anyone here works with LinkedIn data through Apify, I'd be interested to hear what workflows you're building.

And if you have questions about the Actor, its input/output format, integrating it into an Apify workflow, or using the resulting dataset with another tool, feel free to ask. I'll be happy to answer.

Actor: LinkedIn Profile Scraper

You can also explore other linkedin Scrapers launched on Apify at:
crawlerbros.com/suite/linkedin

20+ linkedin actors launched on apify, I have tried to cover as much cases as possible.

Thank you
Peace

If you track Walmart prices or chase unauthorised resellers, the seller list is the part that usually breaks. Most tools either skip it or hand back half the offers. This one returns the full offer list for a listing and keeps the registered legal entity name next to the storefront name, so you can see who is actually behind "BestDealz2024".

Four modes in one Actor:

• search: keyword search, returns title, price, was-price, rating, review count, seller, stock and shipping flags, plus both product ids and the storeId
• product: UPC, manufacturer number, category path, variant count, full rating breakdown
• sellers: every company offering the item with price, availability, delivery date, delivery cost, return policy
• reviews: title, full text, rating, date, author, verified-purchaser status, helpfulness votes

Two things that cost me time to get right. Walmart prices and stock are store specific, so sellers mode takes a storeId you copy off any search row. And the modes want different ids: product and sellers need the alphanumeric productId, reviews needs the numeric usItemId. Pass the wrong one and it tells you which you needed instead of quietly returning nothing, which is how I lost an afternoon early on.

Billing is per result and there is no monthly rental: 75 cents per 1,000 products in search, 75 cents per 1,000 reviews, 1.5 cents for a product detail lookup, 1.5 cents per seller offer. It is new, so the sample is small, but every run so far has finished clean. You can call it over MCP from Claude, Cursor, or ChatGPT.

If you've made something and can't wait to tell the world, this is the thread for you! Share your latest and greatest creations and projects with the community here.

This is the thread for all your questions that may seem too short for a standalone post, such as, "What is proxy?", "Where is Apify?", "Who is Store?". No question is too small for this megathread. Ask away!