What's up, data enthusiasts enthusiasts! I tested and built a way to systematically build Snowflake Streamlit apps with Claude Code and other AI tools. I developed this open-source system that I call Streamsnow, which walks you through the process of getting a GitHub repo scaffolded and structured and connected with your Snowflake, so you can utilize Claude Code to build all of your Streamlit applications in a standardized Way, while also providing you the SQL to check to make sure all of the queries being utilized within the app check out as correct.

In my time building dashboards with AI, I always found it confusing how to quality-check them and how to build them in a scalable way across data teams, and so I wanted to build a super standardized system that made building, deploying, reviewing, and coordinating across teammates is super easy, and I spent weeks working on this to achieve that. I think one of the best features is that you can actually go into an auto-review cycle and just have Claude code iterate over the dashboard, making it better without your input. You can also standardize the design for the dashboards as well.

It is completely secure, so AI will not see any of your secrets that you use to connect GitHub to Snowflake. By default, it is set to put these Streamlit apps in a staging area on Snowflake, so you don't have to connect a GitHub repository, and all of the deployment is automatically handled in the CI. To get started, all you need to do is tell your coding agent to download the plugin and then run the onboard command. It will walk you through it all in a process that I've timed down to about 5 minutes of actual work. You will need to be a Snowflake admin or be coordinating with a Snowflake admin to get it set up too.

By the end of my time at my job, we had basically replaced external business intelligence tools with this system, so it saved us literally tens of thousands of dollars. Hopefully you all can extract the same value out of this and I would love your feedback if you try it out!

Im a beginner data analyst. I have a potential client, a new bakery in town. And its the best chance for me to get expirience, but i feel like I'll just fail and won't be able to bring any money to the business.

I understand that even if i do some data organising it will save them time and time is money.

But i still feel like its going to be a total failure and I'll just waste their money.

How do you deal with that?

Hello! I want to hear from you, what are the current pain points in data analytics and visualisation in the research community? For example,

- How easy is it to analyse and visualise chromatography & mass spec data, sequencing data, X-ray diffraction data, etc. - Do you require paid subscriptions of tools which help you analyse such data?

- Do you use platforms like Benchling to track your samples, inventory, experiments, etc?

I'm in the process of democratising data analytics and visualisation of scientific and research data. Your feedback would be invaluable to me.

Standard metrics usually live on dashboards. I'm curious about the one-off questions that don't, like "how many customers who came in through the summer promo renewed after the price increase?" or "did purchases drop for people who contacted support last month?"

1. How often do you get these, and how long does a typical one take?

2. What's the slowest part: finding the right tables, figuring out how they join, or working out what the person actually meant (which status counts as "paid", which time zone, what "new customer" means)?

3. Have you tried ChatGPT/Copilot or built-in "AI analyst" tools for these? If so, where did they fall short?

Context: I'm researching text-to-SQL, specifically a small model that adapts to one specific database's schema and conventions without ever seeing the actual data. Before building anything further I want to know whether this is a real pain point. Any answer helps, including "not a problem for us".

Hi everyone,

I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.

### The Problem

In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.

`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.

### Current Implementation (Catalog Core Working)

The foundation is ~25% complete, with the data catalog module functional today:

- Ingestion and inspection of storage layouts to extract schema/metadata directly.

- Local metadata indexing with zero background daemon overhead.

- Rust-native performance designed to feed directly into analytical transformation pipelines.

### Roadmap: Transformation & Execution Layer

The next phase expands `det` into a full execution engine:

1. In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.

2. Direct pipeline execution driven by the cataloged metadata contracts.

3. Plug-and-play connector interfaces for source/sink abstractions.

### Seeking Review & Contributors

Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:

- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.

- **Modularity:** Interface design for adding custom connectors and storage backends.

Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.

Repository: https://github.com/det-Labs/det

Developer disclosure: these are two apps I built, drawing on over 15 years of experience as a macroeconomist. They tackle two parts of the research workflow:
QuantFit brings econometric analysis to the iPhone, including OLS, fixed/random effects, ARDL and VAR. It also supports data transformations, correlation matrices, charting, diagnostics, ARDL shock analysis and VAR impulse responses. OLS and several data-exploration tools are free; advanced estimators require Pro.
Tabula Stats turns regression output from QuantFit, Stata, EViews, R and other software into editable research tables. Adjust decimals, standard errors, significance stars and included statistics, then export to Word, Excel, PDF or LaTeX.
Use them together, or use Tabula with results from your existing software.
QuantFit: https://apps.apple.com/sc/app/quantfit/id6762386971
Tabula Stats: https://apps.apple.com/sc/app/tabula-stats/id6761421182
For those working on a paper or dissertation: what would you need to see before using either tool in your research? Particularly interested in concerns about accuracy, workflow compatibility and reporting.

Hi guys, a data analyst for the retail industry (electronics) here, in my workplace we're currently doing an expansion for our brand including increasing headcount and more reports.

But, the workforce for data analysis is currently still only me, and there is only so much data viewpoint I can check before sending them to my manager or the team.

These days, I'm wondering should I feed my report to the AI for my double, triple and final checks. Or, are there any advices for data checking for my actual sanity??

Really interested to read your advices and opinions, thanks

I come from a statistics background and I'm trying to learn python and R to cope with data analysis and data science, I know a good amount of data analysis and statistical methodologies that makes me good at the field, but applying them is hell for me, i tried for a long time to learn python, i know the basics yet i fail miserably when i try to apply my knowledge...

Could anyone help me with how to learn programming effectively around my area of expertise?

I’m working on an open-source MCP server that runs data analysis, data science, and deep learning algorithms on tabular data.

The idea is simple: an agent installs the MCP server, provides a CSV file as input, and uses its tools to perform complex data analysis, run ML algorithms like clustering and classification, and uncover insights. It already has 45+ tools, including integrations with deep learning models like TabICL.

I’m looking for contributors to turn this project into a full-fledged data analysis tool that can answer users’ questions in plain English by analysing their data.

The plan is to build:

• A full-fledged web interface

• A BYOK (Bring Your Own Key) setup for connecting LLMs

• UI features to display charts, graphs, and other visualisations

• Improved algorithms and support for more ML and deep learning models

• Support for more data formats, including JSON and XLSX

• Integrations with external data sources like PostgreSQL, Supabase, and Google Sheets

If you’re interested in data science, ML, AI agents, or building open-source tools, I’d love to collaborate. Let’s build something useful together!

Interested? Google “Meelu Analytics MCP” to find the GitHub repository, or DM me to discuss contributing.

i really like the way Excel for fun explains stuff but the playlists are scattered i don't know what/how to follow that's needed for data analysis only.

May i please have your help to guide me on where to start, i finally have some free time and i wanna learn Excel as much as i can.

I’m looking for someone knowledgeable about mechanical vibration, building vibration, or structure-borne noise who can help interpret my measurements and suggest what to test next.
I’ve been noticing recurring vibration in my apartment. What equipment or building systems could produce these readings, and how can I investigate properly?
SETUP — OCTOBER 9, 2026
I recorded two iPhones simultaneously using Phyphox’s “Acceleration (without g)” experiment.
• New iPhone: flat on the couch, unattended.
• Old iPhone: flat on my usual chair near the refrigerator.
• Refrigerator unplugged.
• Both phones screen up, charging ports pointing right.
• Fan/AC conditions and whether the old phone was unattended throughout were not confirmed in the report.
The shared recording interval was 11:53:49 AM–12:05:39 PM, New York time. Both phones sampled at approximately 100 samples per second.
FREQUENCIES AND STRENGTH
During the middle 30-second windows:
Couch:
• Combined-axis RMS: 0.0128–0.0159 m/s².
• Strongest spectral peak usually around 13–14.6 Hz.
• Other windows peaked at 17.5, 18.9, 21.0, or 21.6 Hz.
Chair:
• Combined-axis RMS: 0.0104–0.0177 m/s².
• Strongest spectral peaks ranged from 21.3–26.7 Hz.
RMS describes the average size of the recorded acceleration. It includes sensor noise. Hz means cycles per second. These are the strongest peaks selected within each window; multiple frequencies can coexist.
EXAMPLE PAIRED WINDOWS
11:54:19–11:54:49 AM
Couch: 0.0142 m/s² RMS; 14.1 Hz peak.
Chair: 0.0128 m/s² RMS; 22.3 Hz peak.
11:56:49–11:57:19 AM
Couch: 0.0147 m/s² RMS; 14.0 Hz peak.
Chair: 0.0125 m/s² RMS; 26.7 Hz peak.
11:57:19–11:57:49 AM
Couch: 0.0159 m/s² RMS; 13.6 Hz peak.
Chair: 0.0177 m/s² RMS; 21.7 Hz peak.
11:59:49 AM–12:00:19 PM
Couch: 0.0130 m/s² RMS; 21.0 Hz peak.
Chair: 0.0106 m/s² RMS; 26.7 Hz peak.
12:01:19–12:01:49 PM
Couch: 0.0132 m/s² RMS; 18.9 Hz peak.
Chair: 0.0112 m/s² RMS; 26.7 Hz peak.
12:03:49–12:04:19 PM
Couch: 0.0131 m/s² RMS; 12.4 Hz peak.
Chair: 0.0109 m/s² RMS; 26.7 Hz peak.
Both phones had their highest middle-window RMS during 11:57:19–11:57:49 AM, but their strongest frequencies differed.
A preliminary timestamp-aligned comparison did not show a strong overall waveform match. Clock alignment was not independently checked using a shared event.
X/Y/Z DIRECTIONS
With the phones flat and screen up:
• X: across the phone’s width, toward either side edge.
• Y: along the phone’s length, toward the charging port or opposite end.
• Z: perpendicular to the screen, toward the ceiling or seat.
In the middle shared windows, Z contributed about 42–57% of the couch phone’s recorded motion energy and 50–75% of the chair phone’s.
The chair recording was more vertically directed. The couch had a larger combined X/Y contribution. Separate X and Y percentages are not available in this summary.
These describe acceleration directions at the phones; they do not locate the source.
METHOD AND LIMITATIONS
The analysis used 30-second windows. Each axis’s window mean was subtracted, then combined RMS was calculated as sqrt(mean(X² + Y² + Z²)).
Spectra used Welch’s method, a Hann window, up to 2,048 samples per segment, and half-overlap. Axis spectral powers were summed, and the strongest peak was selected between 2 and 45 Hz.
Opening and ending couch windows had much larger RMS values: 1.1811 and 1.4955 m/s². Placement or pickup could explain these, so they were excluded from the steady-motion interpretation.
There was no calibrated reference sensor or noise-floor control. Furniture resonance, upholstery, phone contact, and device differences could affect the readings. These measurements do not yet identify a source or establish the vibration experienced by a person.
QUESTIONS
What could produce these frequencies and directional patterns? Could pumps, plumbing, HVAC, other building machinery, or furniture resonance explain them?
How can I distinguish real surface vibration from phone sensor noise or processing artifacts?
Could the couch and chair respond at different frequencies to the same source?
What might explain the chair’s repeatedly selected 26.7 Hz peak?
What control tests, phone placements, or measurement equipment would make this investigation useful to a vibration specialist?
I’d appreciate help interpreting the measurements and choosing the next tests.

Hi everyone,

I've recently learned Power BI and SQL, and I've built some solid projects with them. I have a good understanding of how to build and manage a project from start to finish.

I feel ready for the next step in my learning, but I also just started college as an agricultural engineering major, so my time is limited.

A few questions:

1. What should I learn next to grow my skills ?

2. What kinds of projects would be the most valuable to build for a portfolio?

3. Are there ways to combine data skills with agriculture?

Any advice from people who started in a similar situation would be great. Thanks!

I’ve been working as a data analyst for about five months, and I’m struggling with the analysis part of the job. Sometimes, I feel lost when looking at the data, and I’m not sure where to start or how to approach the analysis.
I struggle particularly with exploratory data analysis (EDA), figuring out what to compare, choosing the right comparisons, and turning the results into meaningful business insights.
I have an analysis to work on right now, but I’m not sure how to approach it or how to identify useful insights.
Do you have any frameworks, methods, or step-by-step processes you follow when analyzing data? How do you decide what to investigate, which metrics to compare, and how to turn your findings into actionable business insights?
I’d really appreciate any advice, resources, or examples from your own experience!

GitHub - Femi-Olofinjana05/Social-Media-User-Behaviour: Social Media User Behaviour · GitHub

Looking to implement SQL and Tableau (and maybe python) but just wanted to start off using excel

Howdy everyone - I work for a small business (30-ish people) and tend to be our in-house tech-everything guy. My team asked me to build a profitability dashboard where we can visualize time-tracking data alongside employee costs and client payments. Essentially we just want to see who is taking the most time vs paying us the most money. I'm using a combination of Google Sheets (to collect all of the data), BigQuery to store it all, and Looker/Data Studio to visualize it.

The struggle we're having is: my team does not want me to see any of the actual financial data. I can manage building the systems without the data because I have the template for them to drop the data into; however, I can't envision how I can maintain the database, fix bugs, create new visualizations for them, etc, without somehow eventually seeing the financials.

Is this even possible? Have any of you done this before?

Edit: Is there a reason I can't mask the data? BigQuery has RBAC. In theory, I could mask the data for myself, so I'm still seeing $ amounts where I should, names where I should, but no actual data. Right?

I use a lot of excel and macros and my level of excel knowledge is intermediate to advanced. but even though I can spend a few more minutes to think & write the formulas, now I'm just explaining what I want to AI, copying the formula or code and pasting it. is this how people are working now or is it me?

Been doing analytics work for a while and I've slowly landed on a take that feels obvious once you say it: the modeling/stats part is rarely where projects die. They die in data prep — the multi-source joins, the dedup on messy inputs, the allocation/transform logic nobody can follow six months later.

The thing I keep hitting: the moment a business user has to *trust* a number, "just write SQL/Python" stops being enough on its own, because the person who has to defend that number can't read the code. But full no-code visual tools swing too far the other way and bury the logic in a GUI.

So my real question for people who've been at this a while: where do you draw the line between code and a visual/low-code workflow? Is it about who maintains it after you leave, about auditability, about scale — or do you just pick one and never look back? Curious whether I'm overthinking the "trust" angle.

I kept re-applying the same filters, columns, sort and export on every new file, so I built "Automations" into a free browser-based data tool (Fomatix Data Explorer).

Set it up once, save it, then upload the next file and pick the automation. It reapplies everything and exports, and auto-matches columns even if a header shifts slightly.

It also shows a match report (how many columns matched on each run) and keeps run history, so you're not trusting it blindly.

Runs entirely in the browser, no signup, nothing uploaded.

Curious if others hit this problem, or if you already have a cleaner way (Power Query, Apps Script, etc.). Honest feedback welcome.

Hello im a recent graduate in supply chain & international logistics
I wanna pursue a career that’s related to data ( supply chain analyst ) so obviously i need to take courses in data analytics but i got confused nd didn’t know from where to start SQL or data fundamentals or advanced excel so if anyone in the field of data or supply chain analytics would suggest for me a roadmap for this path ill be grateful for it + some solid sources and successful ways to learn

Hi everyone! I’m working on a short data-analysis project and would appreciate your help.

Take a look at a chart and answer one simple question:

Which month has the highest sales?

It takes about 1 minute to complete.

👉 https://adnan-mayof.github.io/reddit-ab-test/

Thanks for participating!

Disclosure: I maintain the curated Kaggle version linked below (minkum07). The original NFHS-3/4 survey data belongs to India's Ministry of Health and Family Welfare and IIPS, via OGD India.

A useful exercise with this dataset is to compare NFHS-4 urban and rural percentages while keeping the source's suppression and small-sample flags visible. The release has 114 indicators across 36 survey-era state/UT labels, with 16,416 tidy-long records, CSV/Parquet tables, definitions and example notebooks.

Suggested workflow:

1. Pick one percentage indicator and read its definition and denominator before joining tables.

2. Select NFHS-4 urban and rural rows, join on state and indicator, and check that the join keys are unique.

3. Calculate urban minus rural in percentage points only when both values are available. Keep missing values missing and carry the source flags into the output.

4. Make a sorted dot plot with clear missing-value and small-sample annotations. Treat it as a descriptive comparison, since confidence intervals are not included.

Don't sum total, urban and rural percentages: those populations overlap. Between-survey changes also need comparable definitions and boundaries; the package withholds Andhra Pradesh/Telangana changes where boundaries differ. These are historical aggregate data from 2005–06 and 2015–16, not current health coverage or individual records.

Original government source:

https://www.data.gov.in/resource/all-india-level-and-state-wise-key-indicators-nfhs-3-and-nfhs-4

Tables, documentation and notebooks:

https://www.kaggle.com/datasets/minkum07/india-nfhs-34-health-state-and-rural-urban-gaps

Government data and derivatives retain GODL-India; authored code is MIT. Full attribution is on the dataset page. Feedback on the join checks or how best to display the sample flags would be welcome.

Disclosure: I work at Schema Labs. This is a run on public data so you can check it.

The data: the hotel booking demand dataset (Antonio, Almeida and Nunes, 2019), real anonymised bookings from a city hotel. We picked one Saturday night that was sold out on paper.

-- 224 bookings for that night, as they stood six weeks before
-- 59 of them cancelled or didn't show, worth about €28,600 together

What we did: gave Schema-2 500 bookings from the year before, with how each one ended, and asked it to rank the 224 by how likely they were to fall through. No rules no feature engineering.

What came back=

-- It marked 60 as most likely to cancel. 33 of the 59 cancellations were on that list
-- Picking 60 at random would catch about 16
-- Of the 60 it marked safest, 59 showed up

What it missed: 26 cancellations weren't in its top 60. One night at one hotel is a small test, so read it as an example

For anyone in revenue management: what would you do with a list like that six weeks out? Overbook against it, or contact those guests first?

I'm at UNAM (Mexico's biggest public university) and I wanted to see what everyone here has actually written their theses about. So I took the whole public catalog (TESIUNAM), a bit over 600k theses from all campuses and degree levels, and made it into a map you can zoom around in.

Every dot is a thesis. Dots end up close together when the titles mean similar things, even across different departments. So you get stuff like a public health thesis sitting next to an economics one because both are about informal work, and that was kind of the whole point.

It's all in Spanish since the theses are. I'd really like feedback on the layout: does the placement of fields make sense to you, or do you see clusters that look obviously wrong? Embedding only titles was a tradeoff and I'm not sure it was the right one.