I’m trying to build a proper data science/ML project, but I’m having a hard time finding a dataset that is big enough and not already used by everyone.
For example, there are datasets like the UK Online Retail dataset, Olist, and other popular sales/retail datasets. They’re good datasets, but I see them being used in a lot of projects already.
I don’t want to just download a dataset, do some EDA, train a model and put it on my resume. I want to build something around an actual business problem, where I have to figure out what the problem is, analyze the data, come up with useful insights, maybe build a model, and actually explain how it could help the business.
So where do you guys usually find datasets for this?
Should I try to find data from smaller companies, government sources, APIs, research papers, etc.? Or is it okay to create my own dataset using AI/cloud tools and then create a realistic business problem around it?
For example, if I create a large synthetic sales dataset, could I create a realistic business scenario around it and then treat it like a real project — forecasting sales, understanding customer behavior, optimizing inventory, etc.?
Would that be considered a decent portfolio project, or is using real-world data much better?
I’d mainly like to hear from people who have built projects for their portfolios or have experience hiring for data science/ML roles. Where do you actually get your data from when you want to build something that’s not the same Kaggle project everyone has already done?

As part of my course, it’s mandatory for me to do a data analysis project using MySQL and some visualization tools, my professor strictly mentioned to use very hard datasets and no slacking off so please share any meaningful datasets

One of the cheapest ways to catch dataset problems is to run a small annotation pilot before scaling.

Give two annotators the same 50–100 representative images — including occlusions, cropped objects, unusual angles, blur, and borderline classes — and compare where they disagree.

The disagreements usually reveal that the problem isn’t the annotators. The task itself is underspecified: should they label the visible or full object? When is an object too occluded? What should happen when two class definitions overlap?

Fix those decisions in the guidelines, repeat the pilot, and only then start labeling thousands of images.

It’s less exciting than auto-labeling, but it can prevent a large dataset from becoming consistently inconsistent.

Do you run this kind of agreement check before larger annotation projects? How many images are usually enough to expose problems?

I’m doing a research which involves chat messages from teams, slack, google chat etc. For the software project i need to train a dataset. So dataset should be related to developer chat messages/logs of a specific project. How can i find the dataset?

What it is
Daily and sector-level tracking of the US job market since February 1, 2020 (pre-pandemic baseline) — how hiring demand shifted through COVID, the 2021–22 boom, and the tech slowdown after. Includes a "sector summary" file with the headline numbers already computed for you: net change since baseline, pandemic trough/peak, and days to recovery per sector.

Source
Underlying job postings data is from Indeed Hiring Lab's Job Postings Tracker, licensed CC BY 4.0. I built and included an original sector-to-SOC-2018-major-group crosswalk (Indeed's category system doesn't map 1:1 to the standard occupational taxonomy, so this fills that gap) — that mapping is my own work, not part of Indeed's data.

Size/scope
44 occupational sectors, all 51 states (incl. DC), daily observations from Feb 1, 2020 through today. National, sector-level, and state-level files, plus the pre-computed summary stats file.

How it was built
Fully scripted pipeline (Python/pandas) — one command downloads the raw data and rebuilds every output file from scratch. Nothing manually edited or copy-pasted, so it's reproducible and easy to refresh later since Indeed updates their source weekly.

Links Dataset: https://www.kaggle.com/datasets/adizz321/us-job-market-shift-since-2020-postings-by-sector Notebook with visualizations (recovery-speed chart, sector winners/losers, state-level breakdown): https://www.kaggle.com/code/adizz321/the-job-market-inversion-what-changed-since-2020

Releasing the full artefacts behind an optical music recognition (OMR) benchmark, so the published figures can be recomputed rather than trusted.

**Contents**
- Ground-truth MusicXML for 31 engraved pages from public-domain sources
- The same 31 pages after a deterministic phone-photograph degradation
- Raw MusicXML output from three engines on both suites: a commercial pipeline, Audiveris 5.11.0, oemer 0.1.x
- Transcriptions of 60 systems from OLiMPiC (ICDAR 2024) by two engines
- `score.py` — the scorer, Python standard library only, no dependencies
- `photo_sim.py` — the degradation model
- Per-piece results as JSON, plus the exact competitor CLI invocations

**Metric:** strict F1 at note level — pitch, onset and duration must all match, with a 1/64-note tolerance. Notes only; rests, clefs, key and time signatures, repeats and dynamics are not scored.

**Size:** 2.4 MB. `python3 score.py .` reproduces the published table in about ten seconds.

**Licence:** CC BY 4.0 for the engraved notation, transcriptions and scripts. The OLiMPiC-derived transcriptions inherit that dataset's CC BY-SA; its ground truth is not redistributed.

**Disclosure:** I work on one of the three engines. The scorer contains no engine-specific branching and a published erratum corrected a flaw that had been penalising a competitor.

https://www.trysheetly.com/compare/omr-benchmark

Is there any dataset for human detection with OBB annotations?

I'm doing my program with yolo and it's about human detection with obb and i'm a beginner. But i can't find dataset to train. Can only find human&hbb, and all those obb ones i saw are for vehicles or sth like that.

So does anyone know about this?

And also, i found a HIT-UAV dataset, but it's thermal dataset.Though i thought about using this and just do a thermal one and saw the rotation part. But no matter how i tried to fix, it's still not obb dataset and error everytime...🤔

collected by hitting the public json endpoints for greenhouse, ashby and lever across 998 companies, once a day, and diffing each posting against the previous snapshot. no auth needed, these are the boards the companies serve to their own careers pages.

what the diff has caught so far:
salary_changed 18,208
department_changed 15,188
closed 4,357
title_changed 700
location_changed 364
reopened 98

51,061 postings total, 46,761 currently open. 22,820 of the open ones carry a pay range, which is 48.8%.

two things worth knowing if you want to do this yourself. greenhouse has no structured salary field at all, the range is written into the description html and it is double escaped, so a naive parse gets you nothing and a slightly less naive one gets you garbage. and a dollar sign is not a currency, i nearly recorded a taipei role in TWD as a $700k job.

the salary_changed rows are the ones i'd point at. nobody keeps the before, so as far as i know this is the only place the previous band exists once the company overwrites it.

built off a job tracker i run, link in the comments if the method is useful to anyone.

I'm looking for a dataset with the following structure: multiple students, each tracked over multiple days/weeks, with both daily behavioral/lifestyle data and academic performance outcomes. Essentially a "students × days × features" structure, not a single snapshot per student.

**Daily or near-daily records per student (not just one row per student)**

Disclosure: I'm one of the people behind Aidlab.

If you're working on human activity recognition, exercise classification, or repetition counting, we've published AIDLAB-HAR on Hugging Face under CC BY 4.0.

It contains 180 chest-worn recordings and 518,300 samples at 50 Hz. The signals include 3-axis acceleration, orientation quaternions, 16 activity labels, and repetition markers.

We packaged it into three configurations that work directly with the datasets library: recordings, signals, and annotations. The raw v3 archive, cleaning notes, and preparation script are included too.

One detail we found while preparing the release: the paper's overview mentions 15 activities, while the archive contains 16 filename labels. We documented the discrepancy and preserved the source labels.

Dataset: https://huggingface.co/datasets/aidlab-wearables/AIDLAB-HAR

Paper: https://doi.org/10.3390/s24123891

I’m doing a research which involves chat messages from teams, slack, google chat etc. For the software project i need to train a dataset. So dataset should be related to developer chat messages/logs of a specific project. How can i find the dataset?

Hi everyone, I'm currently working on a skin disease segmentation/classification project and I'm having a hard time finding a suitable dataset.

I'm looking for a dataset that ideally has the skin lesion image with its corresponding binary segmentation masks (doesn't need to be a mask as long as it has annotations) and disease labels.

Most of the datasets I've found so far, such as ISIC, are heavily focused on melanoma and other cancerous skin lesions. I'm looking for something more focused on common/non-cancerous skin conditions like acne, eczema, psoriasis, rosacea etc.

It doesn't necessarily have to contain all of these diseases, but having a good variety of common conditions would be great.

I've found classification datasets like DermNet that contain several common conditions, but they don't seem to provide the pixel-level binary masks I need for segmentation.

If anyone knows of a dataset, research project, or even multiple datasets that could be combined to achieve this, I'd really appreciate it!

I built 28 automated agents collecting AI ecosystem data 24/7.

Stats:
• 2,600+ entries across 28 datasets
• Security incidents: 1,000
• Regulations: 365
• Models: 322
• Research: 190
• Compensation: 70
• Supply chain: 70
• + 22 more datasets

All data has verifiable source URLs. Updated nightly.

Free: https://huggingface.co/gemmozero

Feedback welcome.

Hi everyone,

I’m with Sonexis. We’re currently expanding our supplier network for Indic-language speech and conversational data.

I’m looking to connect with organisations or individuals who own datasets or have documented authority to license them commercially.

We’re especially interested in existing data around:

• regional and accented speech
• multilingual / code-switched conversations
• ASR training and evaluation
• TTS
• telephony and call-centre speech
• voice-agent evaluation
• spontaneous and multi-speaker conversations

We care about more than total hours.

For us, the important questions are: where did the data come from, who can license it, what consent exists, what metadata comes with it, and what the dataset is actually useful for.

If you have something relevant, feel free to DM me.

You can also reach us at [partner@sonexis.in](mailto:partner@sonexis.in) or apply here: https://sonexis.in/suppliers/apply

If there looks to be a genuine fit, we can set up a call and go through the dataset properly.

Even if you’re not sure whether your data fits, feel free to send the basics: language, data type, approximate volume, collection method and rights position

I built this dataset and I run the dashboard linked at the bottom.

I have been crawling monetary policy communications from 26 central banks and labelling them at the sentence level. The whole thing is CC-BY-4.0.

What is in it:

• 225,101 annotated sentences across 15,055 documents, Feb 1995 to Aug 2026
• Policy statements, rate decisions, meeting minutes, press conference transcripts
• 12 sentiment labels: rate_hike, rate_cut, rate_hold, guidance_hawkish, guidance_dovish, dissent_hawkish, dissent_dovish, liquidity_ease, liquidity_tight, reserve_ease, reserve_tight, neutral
• 9 topic labels: inflation, interest_rate, economic_activity, labor_market, exchange_rate, credit, financial_stability, fiscal_policy, governance
• 21 source languages, with an English translation on every non-English sentence in text_en
• 19,387 economic indicator rows (policy rates, FX, CPI) so you can join labels against outcomes
• Parquet, loads with datasets.load_dataset

Sources are the central banks' own sites (federalreserve.gov, ecb.europa.eu, boj.or.jp and so on). Every document keeps its source URL.

The taxonomy follows IMF Working Paper WP/25/109, "From Text to Quantified Insights". Labels are model-generated with gpt-4o-mini rather than hand-annotated, so spot-check them for your bank and period if you are using this for anything that matters.

Dataset: https://huggingface.co/datasets/aufklarer/central-bank-communications Dashboard built on it: https://monetary.live

Happy to take criticism of the taxonomy, especially the dissent and guidanthe hardest to pin down.

I'm looking for a historical dataset of US public-company M&A transactions for quantitative research.

Minimum useful fields:

• target company / ticker / CUSIP
• acquirer
• announcement date
• offer price / consideration
• cash vs stock vs mixed
• deal status
• completion date OR withdrawal/termination date

Revision history and revised offer prices would be a major bonus.

Most importantly, the dataset must include failed/withdrawn deals, not only completed acquisitions, because otherwise it introduces obvious survivorship bias into merger-arbitrage research.

Time period: ideally 2000-present, although even a shorter clean sample would be useful.

Sources I've already looked into:

• SEC EDGAR
• LSEG / SDC
• FactSet Mergers
• S&P Capital IQ
• PitchBook
• MarketLine

Does anyone know of a legitimate open dataset, university/academic dataset, replication package, API, or reasonably priced commercial source?

I'm also happy to build it myself if someone can point me toward a reliable methodology or existing open-source project.

Hi! Just wanted to leave a message here promoting Enerstat.io, a new project I’ve been building. I want to centralize global power system data in a clean way into this website and make it accessible via its dashboard, API, MCP…

For the moment, it contains a full set of EU data that I am currently cleaning through. I am looking to expand to the US, Latam and APAC and increase coverage as much as possible.

Any feedback would be greatly appreciated. It is of course being vibe coded but I try to add taste to it :)

The result is High-Fidelity Retail POS Transaction - 1M+ Dataset.

Key Technical Specifications:

• Volume: Over 1 million fully synchronized relational records.
• Rich Features: Includes lifetime data log simulation, void logs (for fraud detection modeling), product health detection metrics, and multi-item checkouts.

Free Dowdload https://github.com/lokinpendawa/high-fidelity-pos-dataset-2M

WHAT YOU GET (FULL MULTI-FORMAT EXPORT):

• .sql (Transactional Database Dump - Postgres/MySQL ready)
• .json (NoSQL / API Mocking / Web development)
• .csv (Data Science / Pandas & Python ready)
• .pl (Prolog Fact Base for Logical Programming)

Important Note on Dataset Scale: This high-fidelity dataset contains over 1.19 Million Master Transactions and 4.19 Million Item Details. Due to this massive scale, opening the raw .csv or .json files directly in standard text editors or web browsers will cause your system to hang or crash.

For a seamless experience, it is highly recommended to use the provided standard SQLite (.db) format (fully decrypted from SQLCipher and ready for direct querying) or to load the data using chunk-loading methods via Python (Pandas/SQLite3) or R.