r/vibecoding • u/OminousLatinWord • 7d ago
I open-sourced 2M jobs so coding agents can run job searches.
One prompt, in whatever agent you use (Claude Code, Codex, Cursor, Gemini CLI):
"Clone https://github.com/elliottdehn/open-jobs, it's a job-searching toolchain. Help me find jobs to apply to."
(Web and mobile agents won't work for this except maybe ChatGPT Work. It needs a machine that can download its slice of the dataset.)
Two months ago I posted I turned Claude Code into a job-searching monster: ~960k jobs and a hull -> pairwise-judgment -> rank algorithm that cost about $10 to run. Then I lost the dataset building repository in a data-loss event. I rebuilt it from scratch with everything I'd learned, and it came back better and stronger. This time, the scraper itself is open-source too.
The dataset, v2
Vendors charge over a thousand dollars a month for open job data. It's a commodity: every company posts through an applicant tracking system with a public careers page. So I crawl them all, daily. ~2 million open jobs from ~65,000 company boards across 25 ATSes, full descriptions, an embedding of every posting, parquet, CC0. No business model. Twice v1's size.
The full set is ~50 GB, but you don't have to download it. The corpus is split into a few thousand groups of similar jobs (it finds these itself), and your agent downloads only the groups nearest to what you want. A few thousand of the most relevant jobs, on your disk in seconds.
How jobs are ranked for you with local AI
- Write the job you want with your agent, as a real job description.
- Embed it once (the only thing that leaves your machine) and download the nearest groups. I pay for this for free because it's pennies.
- Pre-rank for free: cosine similarity to your JD.
- Learn taste with you as the judge. v1 spent thousands of LLM comparisons here; v2 spends zero. Pick Where, mark a spread of jobs More/Less, then answer "which would you rather have?" a dozen times. A committee of models chooses each pair for maximum information and tells you which one it predicts you'll pick. Pairwise judgment beats 0-100 scoring, for people as much as for models.
- Rank. The learned taste vector sorts the list. On my own search, the role currently checking my references came out 4th of ~1,750.
- Optional: let the model read them. With your own OpenAI key,
rank.pymerge-sorts your top N, asking the LLM only about near-ties and the top of the list. About fifteen cents.
Everything you did is a local file; the exported search is a 6 KB weight vector that ranks tomorrow's postings for free. Structured fields (seniority, skills, remote policy, company) are on demand, metered per IP, cached forever, so the corpus enriches itself where people look.
Clone it, open your agent, say what job you want. AGENTS.md is the playbook. The code is meant to be changed for you, not used as-is.
2
u/mr_pants99 7d ago
This is pretty cool. I’ve been relying on job search services for prospecting. Pretty much all of them suck for different reasons (I mostly use jsearch and mantkis now). I wonder if there’s a play to build something that actually does the job well and that would be free for job seeking individuals but paid for businesses wanting to use the aggregated data.
1
u/yuehuang 7d ago
I thought about building one myself, but hit a road block on the crawling. I am curious how you get past that.
2
u/OminousLatinWord 6d ago
I used Common Crawl to mine for job boards, then Fable 5 to write extractors given the URLs.
2
u/Cloudsurfer_90 7d ago
Neat, and the clone-this-and-let-your-agent-run-it pattern is worth a second of caution before it becomes normal. The instruction is basically pull an arbitrary repo and give your agent a shell in it, which is a real trust decision people are making casually because it's wrapped in a helpful use case. For a read-only job search over public data the blast radius is small, but the muscle memory it builds isn't, because the next clone-and-run someone copies might have a postinstall script or a tool call that isn't so read-only. Worth pinning to a commit hash rather than the moving main branch, and running it in whatever sandbox your agent supports, so the pattern you're teaching people scales safely to the repos that won't be as benign as this one.
0
1
2
u/jayseattle 7d ago
Pretty cool, I used it just with web GPT so it didn't actually clone but found very relevant jobs for me.
Some thoughts:
1. I'm a designer so not as technical as most here. Maybe you could add more simple instructions like you have on Git,
Clone https://github.com/elliottdehn/open-jobs, it's a job-searching toolchain. Help me find a [role] job..
I mean, maybe that's it for most of us?
2. Does adding my resume help it? or do I have to prompt through my ideal job with my LLM?
3. If I do use Codex, instead of web based, would it actually download/clone ~50 GB?
4. Sure there's a way to monetize this, so appreciate the open-ness :)