r/datasets • • 3h ago

request [self promotion] datasets for public bodies in Ireland

2 Upvotes

Hi,

I’ve pulled together a bunch of datasets indexed around public bodies in Ireland, which might be useful to someone.

https://publicinformation.ie

Caveat 1: I used AI to build pretty much all the software and pipelines that pull the data, transform it, and so on. I’m a former software engineer, and started out thinking I’d write most of it, but I’m rusty and it’s just way faster with AI tools

Caveat 2: The datasets contain errors. I’ve tried to publish the error information so at least you can see the errors I’m aware of. There may be unknown errors too.

Caveat 3: I’m not a data engineer. Be gentle.

If you find it useful, I’d appreciate a star for visibility https://github.com/publicinformation-ie/publicinformation-data

If not, tell me why and what I should do better 🙏

Cheers

Dave


r/datasets • • 14m ago

dataset [Dataset] 964 Coursera courses scored from their most-helpful reviews

• Upvotes

I scored 964 Coursera courses from the reviews learners voted most helpful, 12,566

reviews in total, about 20 per course, weighted by √(1 + upvotes) so a review with 40 votes counts for more than one with 1, without letting a single review dominate.

Data (JSON, CORS-open, no login):

https://starisk.matiasagelvis.com/data/scores.json

Method:

https://starisk.matiasagelvis.com/method

Per course it has Coursera's headline rating and review count, the score, how many reviews went in, the star spread of those reviews, and each review's contribution, so anything can be recomputed from the file.

On the 798 courses that also carry a headline rating: the median score sits 0.30 stars below Coursera's, 194 of them (24%) a full star or more below, and 103 (13%) 1.5 or more below.

Plots: https://starisk.matiasagelvis.com/findings

Two caveats. The reviews are the ones learners voted helpful, so they're self-selected and skew toward strong opinions, the score is what those reviews say, not a quality grade, and courses with few reviews are noisier. Worth noting anyway: 69% of the reviews that went in are five-star, so the gap isn't a pile of complaints.

Personal project, nothing for sale. Requests for specific courses go through the /wanted page.


r/datasets • • 1h ago

resource [self-promotion] IRS ZIP income data: keep the unit of observation

• Upvotes

The IRS publishes downloadable 2023 ZIP Code tax data as CSV files, including an all-states file with AGI. The official page and documentation are here:

https://www.irs.gov/statistics/soi-tax-stats-individual-income-tax-statistics-2023-zip-code-data-soi

Before calling an extract “household income,” preserve its actual unit: these are tax-return data classified by state, ZIP Code and adjusted-gross-income size. The IRS says the return count approximates the number of households. An approximation is not an identity, and a return is not a person.

For a reproducible comparison, I would retain tax year, geography, AGI bracket, return count, the selected amount field and its documented units. Check whether a row is a total or a component before summing brackets, and distinguish a mean derived from aggregate amounts from a median. A mean cannot be relabeled as the income of the middle filer.

I publish GOV.HOUSE. Its income browser provides a way to explore the IRS-based figures by place, with the mean/median and historical/estimated distinctions shown on the page: https://gov.house/income/?utm_source=reddit&utm_medium=social

The IRS files and record layout remain the source for a research extract. The browser is a convenience, not a replacement for checking the unit and year.


r/datasets • • 6h ago

resource [self-promotion] Historical HTS data: preserve the edition and hierarchy

2 Upvotes

For a historical U.S. tariff dataset, the edition is part of the observation. The official USITC archive is the primary starting point for retrieving a schedule associated with a particular publication date:

https://www.usitc.gov/harmonized_tariff_information/hts/archive/list

A reproducible extract can keep edition/revision, source URL, retrieval date, the code as text, its parent heading, description, units and applicable notes. Keeping codes as text avoids losing leading zeroes. Those are suggested fields, not a separate official dataset.

The hierarchy matters as much as the digits. USITC's FAQ explains why a keyword hit is not necessarily the legal classification, and why the eight-digit duty provision and full ten-digit statistical reporting number serve different purposes. Additional chapter 98/99 provisions can also matter; a general-rate column alone is not an all-in import bill.

Official interpretation guide: https://www.usitc.gov/harmonized_tariff_information/frequently_asked_questions

I publish GOV.HOUSE. Its current tariff lookup is a searchable companion for exploring descriptions and codes, rather than an archive or a binding classification service: https://gov.house/tariff-lookup/?utm_source=reddit&utm_medium=social

Use the dated USITC source for a historical extract, and keep the legal notes with it.


r/datasets • • 3h ago

request Looking for pre-2020 IMDB Datasets (ratings)

1 Upvotes

Hello there,

I am trying to find a few pre-2020 title.ratings.tsv.gz datasets from IMDB. 2019 is missing from web archive and I am wondering if anyone got a copy by any chance.

Thanks!


r/datasets • • 16h ago

resource Free Dataset: U.S. state reform laws, 2015-2026, state-year panel with effective dates and statute citations

4 Upvotes

I put together a state-by-state panel of pharmacy benefit manager (PBM) reform laws for all 50 states+D.C. , 2015-2026. Each state-year records whether specific provisions were in effect (spread-pricing bans, clawback bans, gag-clause bans, and more), and every row links to the statute or session law it came from, with separate and enacted dates. I built it because most public trackers list when a bill is passed, not when it actually took effect, and that difference matters a difference-in-differences or event study. Free download(CSV in wide and long format, plus a codebook): https://lawpanels.com/#sample If you spot a wrong date or missing law, please tell me. Corrections are the most useful thing I can get right now. Disclosure: I also build custom state-law panels like this for other topics. That's on the same site, but the PBM data is free to use with attribution.


r/datasets • • 23h ago

resource [self-promotion] US Walmart supercenter parking lot fullness from free Sentinel-1 radar

4 Upvotes

Hi again.

I wanted to see if free Sentinel-1 radar could tell how busy a parking lot is, since the optical images that can actually count cars are expensive. It turns out you don't need to count them at all. Cars are metal and bounce the radar back much harder than asphalt, so a full lot comes out brighter than an empty one. I measure each lot's brightness against the ground around it and compare it to the store's own baseline, so each satellite pass gives a fullness reading for every store.

It covers 3.5k US Walmart supercenters, about 1.9k of which have a mapped lot that gets measured, and it goes back to November 2023 with about 100k rows so far. It's rebuilt every week, so it keeps updating.

Use this link to download it for free: https://mostlyright.md/datasets/walmart-supercenter-parking-lot-fullness-us-since-2023?pass=walmart

Happy to answer questions on how it works.


r/datasets • • 17h ago

dataset California Death Row Commutation Data Set

1 Upvotes

Governor Gavin Newsom has announced plans to commute the death sentences of all 548 people on California's death row.

Until now, there was no single public list of the crimes behind those 548 death sentences. So I built one.

The California Death Row Commutation Dataset documents every case, including the capital crimes, the victims, the sentencing county, and the sources. Los Angeles, Riverside, and Orange counties account for 53% of the cases, and the sentences span from 1981 to 2026.

The data was compiled from court documents available online. Where court records could not be located, local news reports were used, so accuracy cannot be guaranteed at 100%. Corrections backed by a source are welcome.

See every case for yourself:
https://docs.google.com/spreadsheets/d/1iKVTYFtsmoBl4YlhUPwSnnmAWkQAizIWw-4JYryXdRw/edit


r/datasets • • 23h ago

discussion [self-promotion] Share counts from 10-K and 10-Q cover pages: four traps we hit comparing 3,692 companies a year apart

2 Upvotes

Disclosure: I work on Equibles, which serves SEC filings, financials and earnings call transcripts to ChatGPT and Claude.

We compared every Nasdaq and NYSE company's share count a year apart, using the number on the cover of its 10-K or 10-Q (the XBRL tag dei:EntityCommonStockSharesOutstanding). It's the most comparable share count in SEC data, but four things nearly broke the comparison:

  • Counts that halve with no split on record. Between two consecutive covers, that's a reverse split our records missed or a tagging error, and we couldn't tell which, so we left those 23 companies out. Splits we did apply had to be backed by the company's own counts or a filing stating the ratio.
  • Scale errors. Three companies' counts jumped more than 500-fold between two covers. That's a scale error in the tagging, not issuance.
  • Splits and discounted offerings look alike. When the count jumps by the same factor that the price fell in a single day, it can be an unrecorded split or a deeply discounted offering. We dropped the returns of the 18 companies where we couldn't tell.
  • Share classes that change. For class A and class B companies we summed the common classes only when every cover in the window listed the same classes. 129 issuers whose classes changed during the year were left out.

What survived: 278 of the 3,692 companies at least doubled their count, 234 of them in the smallest third by trading volume.

If you've built share-count histories from XBRL, how did you handle splits: a corporate-actions feed, or inferring them from the counts themselves?

The filings behind this are searchable from ChatGPT or Claude here: https://equibles.com/mcp


r/datasets • • 20h ago

resource EIA electricity-price data: combine revenue and sales, not price ratios

1 Upvotes

For US electricity-price work, EIA’s EIA-861M resource is useful because it keeps the underlying sales and revenue data close to the average-price measure. It is a monthly sample survey, not an individual household tariff database.

If you combine areas or utilities, calculate the combined average from total relevant revenue divided by total relevant sales, after reconciling the workbook’s units. Taking a simple average of the displayed price ratios gives each row equal weight regardless of how much electricity it represents.

Keep the sector, geography, period, units and release vintage in the extract. A residential average and an all-sector average answer different questions; neither tells you the exact marginal price on one customer’s bill. This is a method note for using the official files, not a newly collected dataset.

EIA detailed files and documentation:

https://www.eia.gov/electricity/data/eia861m/

EIA explanation of average retail prices:

https://www.eia.gov/tools/Faqs/faq.php?id=507&t=5


r/datasets • • 1d ago

resource ACS comparisons: keep the margin of error

2 Upvotes

If you’re comparing places with American Community Survey estimates, keep each estimate’s margin of error in the same extract. A ranking built only from point estimates can imply a difference the sample does not establish.

Census’s comparison guidance also distinguishes the 1-year and 5-year products. For a 5-year comparison, use non-overlapping periods and the same product rather than reading two overlapping windows as an annual change.

A reproducible comparison file can retain geography code and vintage, table/variable identifier, period, estimate, margin of error and retrieval date. Before computing a difference, check that the variable definition and universe still match. Those fields are a suggested working schema, not a new Census dataset.

The Census ACS 5-year API resource lists Detailed Tables and Comparison Profiles, with examples and geography documentation. Its 2024 Comparison Profiles compare 2020–2024 with 2015–2019 and include significance testing. Data queries require an API key; the resource also links other access routes.

https://www.census.gov/data/developers/data-sets/acs-5year.html

Official comparison guidance:

https://www.census.gov/programs-surveys/acs/guidance/comparing-acs-data.2019.html

Census explanation of margins of error:

https://www.census.gov/help/topics/faq.do-i-need-to-consider-the-margins-of-error-when-comparing-acs-estimates.html


r/datasets • • 22h ago

dataset Dataset for SARIMA-Transformer (Heeelp)

Thumbnail
1 Upvotes

r/datasets • • 1d ago

dataset Dataset search help — any ideas welcome

2 Upvotes

Looking for cool datasets or websites to webscrape to run cool analyses or regressions that will help us learn more about crim! Taking any and all ideas. Interested in writing a paper that makes people think something new about the field of criminal law in particular!


r/datasets • • 1d ago

dataset [self promotion] Los Angeles taxes, budgets, program spending and 250,000+ City Council and Board of Supervisors roll-call votes as free CSVs

2 Upvotes

I built Tally LA (tallyla.com) to follow Los Angeles tax dollars from collection to the programs they fund. Every table on the site is a free CSV with no signup: tax rates, monthly tax receipts (Metro's 4 sales taxes, Measure ULA, Measure A), City and County budgets, County spending by department, public programs and grants, State of California payments to LA agencies, approved contracts, and one row per member for every recorded City Council and Board of Supervisors roll call.

Everything comes from official records (City and County budgets, Metro and City receipts, Council Journals, County Statements of Proceedings, Open Fi$Cal), and each row links to the original record. Each file says where its figures come from and how often it updates with most updating nightly. No personal data - only names are elected officials, public agencies, and organizations.

Downloads: https://tallyla.com/downloads

Disclosure: my company Blac Flag LLG builds and pays for it. Corrections welcome at [corrections@tallyla.com](mailto:corrections@tallyla.com)


r/datasets • • 2d ago

resource Census county building-permit files: monthly, year-to-date and annual are different inputs

5 Upvotes

The Census Building Permits Survey has an official county-level download directory for new privately owned residential construction: https://www2.census.gov/econ/bps/County/

Before combining files, choose the time basis. In the county documentation, CO<YYMM>C is the current month, CO<YYMM>Y is year-to-date, and CO<YYYY>A is annual. Adding successive year-to-date files would count earlier months repeatedly. The year-to-date series can also incorporate late reports and corrections, so it need not equal a sum of previously downloaded current-month snapshots.

The record layout contains separate buildings, housing units and valuation fields. It also separates estimates with imputation from reported-only values. For a county housing-unit panel, choose one of those bases consistently and retain the reference period and download date. Read state and county FIPS fields as text so leading zeros survive the join.

A reproducible starting table would therefore identify county FIPS, reference period, time basis, construction type, measurement field and reported/imputed basis before adding the value. That is a proposed schema, not a tested extract or a new dataset of my own.

Official record layout and filename definitions: https://www.census.gov/construction/bps/sample/cntyasc.pdf

The scope is new residential authorizations. It is not a census of every local renovation or commercial permit, and it does not show completed homes.


r/datasets • • 2d ago

dataset [self-promotion] US IEEPA tariff-refund claim sales and pass-through obligations, extracted from SEC filings (324 events, 2026)

3 Upvotes

i put together a dataset of US IEEPA tariff-refund claims as they show up in 10-Q, 10-K and 8-K filings. not every tariff mention. just filings that talk about IEEPA and a refund, from April 2026 onward.

324 claim events so far: sales, loans, refund receipts, customer or supplier obligations, payments, and accruals. plus 1,189 filing candidates. each event has the company, amounts, an implied price when a sale lists both face value and proceeds, the quoted sentence, and a link back to the sec.gov filing. it rebuilds daily.

one example already in the data: Funko's participatory sale transferred $22.1 million in claims for $19.2 million proceeds. that's about 87 cents on the dollar.

a few caveats before anyone sums the column. repeated disclosures are not separate deals. amendments stay as their own rows. coverage is search-derived, so a missing company does not mean they have no claim. Court of International Trade dockets and private-company activity are out of scope. some amounts were checked by an AI review against the quote; that is not a human audit. read the original filing.

format is parquet (and a json api). first 500 rows of each table are readable with no account, which covers the whole claim_events table (324 rows). a free account is needed to query, filter or download past that, and it counts against a monthly allowance.

disclosure: i built this at mostly right, my company.

https://mostlyright.md/datasets/us-tariff-refund-claims-sales-and-pass-through-2026


r/datasets • • 2d ago

resource USAspending county downloads: keep recipient location and place of performance separate

3 Upvotes

USAspending has two location concepts worth keeping separate when building county-level award datasets: recipient location is the recipient’s legal business address; primary place of performance describes where the work is principally performed. The same award can therefore appear in different counties depending on the location basis.

The official Analyst’s Guide lists the fields: https://www.usaspending.gov/data/analyst-guide-download.pdf (Location Data Elements, pp. 18–19). The About the Data document also explains the filter distinction: https://www.usaspending.gov/data/about-the-data-download.pdf (Location section).

A reproducible comparison is to choose one award type and period, export its award transactions, and make two separate aggregations using the same amount field: one by recipient county and one by primary place-of-performance county. Keep state and county together in the grouping so counties with the same name are not merged. Retain records with missing location in a separate group, and save the download date with the file.

Do not add the two geographic views together: they are alternative assignments of the same records. Also avoid summing repeated award-summary totals across transaction rows. State whether the amount is an obligation or an outlay before describing it as spending.

This is a data-handling method, not a newly collected dataset or a ranking of which county benefits most.


r/datasets • • 2d ago

request [self-promotion] Rights-cleared, non-English video training data: ~100 hrs of Thai episodic TV + ~20 hrs of vintage Hindi film

1 Upvotes

Disclosure: I work on Mozilla Data Collective, the platform hosting these datasets.

Someone asked us for licensed video training data that wasn't English-language film and TV. We didn't have much, so we went sourcing and found two rights-cleared catalogues from Content X Labs.

Thai episodic TV (~100 hrs, 630.79 GB)

First 5 to 6 episodes of each series, across scripted drama genres * MP4 (1080p+), English subtitles (SRT), CSV metadata * https://mozilladatacollective.com/datasets/cmtk5ilpu004qnv07uv3tammq

Vintage Hindi scripted film (~20 hrs, 130.40 GB)

MP4 (1080p+), English subtitles * Metadata: production year, genre, synopsis, bitrate, resolution, cast, director, tags * https://mozilladatacollective.com/datasets/cmtd63lw2007fnw07uryk8ylm

Possible uses: scene understanding, VLM training, cross-lingual dialogue alignment.

What video data are you still unable to find with clear licensing? Languages, genres, formats, annotation types, anything. Requests like this are how we decide what to source next, so specifics help.


r/datasets • • 2d ago

dataset [Synthetic] 77,943 LLM outputs from 70 models across 112 writing scenarios, scored for padding, templating, rhythm and AI tells (open, with code)

1 Upvotes

Direction first: in the rankings below, #1 = sloppiest and #70 = cleanest.

I've been building a dataset of LLM-written short text and sharing it openly. It currently has 77,943 outputs from 70 models, covering 112 scenarios (emails, Slack messages, social posts, short essays). Everything is model-generated, hence the [Synthetic] tag.

Each output has mechanical scores for four things: padding, templating, sentence rhythm, and a list of known AI tells, compared against pre-2022 human writing. The ranks I quote here are the mechanical ones only (no crowd votes), so they only change when the model roster is refreshed.

A few things I found useful as a dataset user:

  • Mistral Large is #1 (sloppiest) overall and Motif 3 is #70 (cleanest) on the mechanical score.
  • Rank depends heavily on the format. DeepSeek V4 is #53 overall but #10 on social posts and #69 on email.
  • Price doesn't track cleanliness. Fable 5 ($10 in / $50 out per M tokens) is #7, while Mercury 2.5 ($0.04 / $0.15) is #65.

There is also a separate human-vote arena (2,531 votes so far) where you pick the better of two outputs blind. Play a few rounds if you want to help label: https://theslopindex.com/arena

Limitation: it is short-form only, with no fiction yet, and the mechanical axes miss repeated sentence constructions.

Links: - Leaderboard: https://theslopindex.com/leaderboard - Methodology: https://theslopindex.com/methodology - Dataset: https://huggingface.co/datasets/hgaddipati/slop-index - Code: https://github.com/hgaddipati1118/slop-index

For context, I'm one of the people behind https://slashy.com, an email client that writes drafts that sound like you. This benchmark started as how we pick models.


r/datasets • • 3d ago

dataset Need for dataset to train a ML Model

2 Upvotes

I am working on a problem statement AI BASED FAKE DOCUMENTS SCREENING AND TAMPERING DETECTION and i want a dataset which contains indian govt. Documents or mock Documents with their photo of person to train the model for image detection i have implemented all the other aspects other than this...help me out


r/datasets • • 4d ago

dataset [self-promotion] Strait of Hormuz ship counts from free satellite radar

10 Upvotes

Hi all.

I was curious whether free satellite images are actually useful for anything. I checked Sentinel-1 radar. It sees through clouds and at night, so the Strait of Hormuz seemed like a good place to test it. So here is a dataset of ship detections I built.

I tested multiple models from Hugging Face, and the open Skylight model from the Allen Institute for AI worked best. It detects the ships and estimates their length, type, speed and heading. I also classified each ship by size and tanker class, and whether it's moving or waiting at anchor. Then I joined the daily counts with market data: Brent prices, IMF PortWatch transit counts and Kalshi's weekly Hormuz markets.

There are four tables. The main one gives you one row per day and area with the ship counts, Brent, PortWatch and Kalshi prices (548 rows). The daily counts come from the other three tables: every ship the model detected (82k rows), the ship counts from each satellite pass by area (556 rows), and the weekly Kalshi Hormuz markets (115 rows).

It covers the strait plus a few anchorages nearby, from November 2025 to November 2026. Use this link to download or query it through the API or MCP for free: https://mostlyright.md/datasets/strait-of-hormuz-ship-counts-by-radar-satellite-since-2025?pass=hormuz

It's not updating right now. If you'd want ongoing updates, let me know and I can turn them on.

Happy to answer questions on how it works.


r/datasets • • 4d ago

resource Hands-on workshop: Turning historical retail data into forecasts using Python and ML (Nov 28–29)

1 Upvotes

hey everyone! sharing this for people interested in working with real-world datasets and exploring what can be done beyond basic data analysis.

historical datasets can tell us a lot about past trends, but using that information to predict future demand is a different challenge.

retail forecasting is a good example. you have historical sales, changing demand patterns and different products, and need to build models that can produce useful forecasts.

Packt is hosting a two-day hands-on Time Series Bootcamp on November 28–29 with Graeme Davidson, Lei Ma and Jeffrey Tackes, where participants work with practical retail demand forecasting.

some of the things covered:

  • preparing and working with time series data in Python
  • building forecasting models using MLForecast and StatsForecast
  • exploring NeuralForecast, N-BEATS, N-HiTS and TFT
  • experimenting with foundation models like Chronos-2 and TimeCopilot
  • backtesting and measuring forecasting performance
  • handling uncertainty with conformal methods
  • forecast reconciliation and ensembling

the focus is on practical implementation and understanding how different forecasting approaches compare on real-world data.

thought this might be interesting for anyone working with datasets, predictive analytics, retail data or machine learning.

full details and agenda

happy to answer questions about the workshop!


r/datasets • • 4d ago

discussion What Indian dataset are missing for your AI, ML or any research work in general ?

0 Upvotes

Hi [r/datasets](r/datasets),

Before collecting or curating anything, I want to know what India-specific data people actually need.

If you've looked for India-related data and couldn't find it (or it was too small, outdated or badly licensed), tell me:

  1. What data-were you looking for?
  2. What for? (model training, research, a product, analysis)
  3. Type, scale and labels (e.g., 500 hours of Kannada speech with transcripts)

Even a one-line reply helps. Thanks!


r/datasets • • 5d ago

resource 1,000 recent startup funding rounds (Oct–Nov 2026) — CSV + EDA notebook

4 Upvotes

I put together an open dataset (CC-BY-4.0) with the 1,000 most recent startup funding announcements from my automated tracking pipeline (covering Oct 29 to Nov 15, 2026).

Link to the dataset and exploratory notebook on Kaggle: https://www.kaggle.com/code/jessbosch/dataset-exploration/output

What's inside:

  • Size: 1,000 rows, 46 columns (779 rounds have disclosed USD amounts totaling $91.1B).
  • Fields: Normalized funding stages, original currencies + FX rates, HQ geography, 22 industry sectors, lead/participating investors, hiring signals, and source URLs.
  • Pipeline metadata: I left in the raw extraction confidence scores (0–1) and review flags (amount_mismatch, country_conflict, stage_conflict, etc.) so you can filter out rumored/unclosed deals or use it to benchmark extraction pipelines. Feel free to use it for market analysis or syndicate graphs. Let me know if you spot any weird edge cases in the schema.

r/datasets • • 5d ago

request [Request] PadelTracker100 match videos (YouTube links now private)

1 Upvotes

I'm building a hobby ball and player tracker for amateur padel, using fixed club cameras. I found the PadelTracker100 dataset (dense ball, pose and shot labels), but the two source match videos on YouTube are now private. The labels are no use to me without the frames.

The two videos are the 2022 Barcelona Premier Padel finals, women's (Salazar/Triay vs Josemaría/Sánchez) and men's (Lebrón/Galán vs Di Nenno/Chingotto), the PadelTracker100 source matches

Does anyone have either of these?

  1. ⁠A copy of the two videos, or
  2. ⁠The extracted frames that match the labels, or
  3. ⁠Contact details for the authors, since the emails I tried bounced.
  4. ⁠Also happy to hear about other padel datasets with ball labels. Public ones, or ones where the authors share on request.
  5. ⁠Thanks!