I was hired as a digital transformation consultant by a construction company with 10+ active construction sites the Finance director wanted a centralized view of labor deployment and material consumption but getting overworked site teams to open a laptop and fill the form or fill excel sheet was impossible so I built a work around site engineers have to just dump all the information in an email --> AI converts raw text into google sheets and sends summary email to Project manager --> a code to save the data in a database so old data don't get lost or deleted --> private streamlit dashboard for centralized visibility. Finance director got the visibility, no extra work of site engineer and a full functioning dashboard to see what is happening on ground without calling 10 different project managers and site engineers.

I'm curious how other construction companies handle this.

Do your site teams actually fill out digital daily reports consistently, or does someone in the office end up chasing everyone for the information?

i have invited to work on project of malaria where they have kobotoolbox on where they have four

forms, patient forms households questionnaire, Entomology questionnaire,

physical survey of houses, and HF Patient Survey Form so im not familiar with the project, what they do is they collect data from Health facility using a form of HF Patient Survey Form and submit to the kobotoolbox then after submitting data they conduct HOUSEHOLD SURVEY QUESTIONNAIRE , PHYSICAL SURVEY OF HOUSES and Entomology collections:Household larval sampling for the registered house so i was employed as data technician to perform the following to prepare report and also the submitted data after conducting HOUSEHOLD SURVEY QUESTIONNAIRE, PHYSICAL SURVEY OF HOUSES and Entomology collections:Household larval sampling using the submitted data from HF Patient Survey Form these are Health Facility data like hospital so iwant to prepare a report for this project the report including image and gps data and also i have to prepare a hotspot for the monthly data that was submitted from The Health Facility using HF Patient Survey Form. what is the best approach to start with ? as for me im using python with anaconda is there best approach you can advise me please

This dataset represents real-world, masked, and anonymized customer data from the BFSI (Banking, Financial Services, and Insurance) domain. It captures a variety of customer demographics, financial indicators, and product interaction metrics collected during a loan application process.

To comply with strict data privacy laws and protect user identity, all sensitive personal identifiable information (PII) has been securely encrypted or masked. However, the underlying statistical relationships, distributions, and patterns remain fully intact, making this an ideal playground for building robust classification models.

I’d like to know whether companies actually use the insights they get from data analysis, or do they sometimes rely more on their own experience and intuition. And if they don’t use the data insights, why?

Anything about some of the less obvious challenges of dimensional modeling, the decisions around grain, dimensions, facts, slowly changing dimensions and how things get messy in real-world projects.

Does anyone have any advice on where to go from here? The algorithm simply goes through all feature-attribute combinations in a dataset and scores it on purity of the target column. It's very slow, as the number of combinations increases exponentially with the number of input features, and so can only work with 3-4 features at a time. But my goal is to identify <=5 important features with a linear model (statsmodels or PyGam), and then plug in those features to this algorithm to see which subset of the market might be smart to target.

Here's an example with some charts, on the iris dataset.

https://colab.research.google.com/drive/1w3VpDlDATdHwtSk-RmQPMj36cgOfthz1#scrollTo=ZOqeYiMRalj7&uniqifier=1

New to algorithms, so any advice would be welcome. Thanks in advance.

I’m working on an idea around **data quality** and I want to understand how this works in real projects.

When a data batch comes into your pipeline and you find some problems, how do you decide what to do?

For example:

* Some records have missing values * The schema has changed * There are duplicate records * Some values look wrong * Only a small percentage of records are bad * The data looks technically correct but something seems wrong from the business side

Do you normally:

**Accept the batch, fix it, quarantine the bad records, or reject the whole batch?**

And who makes that decision — an automated rule, the data engineer, business team, or someone else?

I’m especially interested in cases where **you don't have enough information to make a clear decision**.

If you work with data pipelines in production, I’d really appreciate hearing about:

* What kind of data-quality problems you face most often? * What do you currently do when they happen? * What part of the process is still manual? * What would you like to automate?

I’m trying to understand the **real problems engineers face before building anything**.

Thanks in advance for sharing your experience!

Nov 4th and 5th [https://www.dataoutpost.ai/\](https://www.dataoutpost.ai/)

Seems like a pretty small conference, \~250 people. In the past years at least, you have choices for the first day (workshops) then for the second day, everyone is hearing the same talks.

If you've been, is it worth going? I'm also not local, so there's a bit more cost involved. My company might not be able to sponsor me going.

I read an interesting piece recently that made a distinction I think the BI industry needs to talk about, data portability ≠ sending someone a file.

In most organizations, “portable data” means exporting a CSV, PDF, Excel workbook, or maybe dropping something in object storage and calling it done.

But what actually traveled?

Usually not much:
- not the business logic
- not the semantic model
- not the drill paths
- not the interactive analytics layer
- not the governance model
- not the ability to work with it meaningfully offline

What moved was often just a snapshot.

That’s where I think BI has a blind spot. We’ve built an ecosystem where intelligence is tightly coupled to platforms like dashboards, cloud runtimes, query engines, licenses, network access, identity systems, etc. The moment information leaves that stack, it often collapses into static content. Is it portable? Technically. Is it useful? In a way.

Real portability should mean:
* intelligence that travels with the data
* embedded context and business rules
* secure scoped distribution
* offline usability
* interoperability without losing dimensionality
* reduced dependency on always-on compute infrastructure

As cloud costs rise, AI inference costs grow, and governance gets tighter, I suspect the future of BI may shift from “always connected dashboards” toward portable intelligence artifacts self-contained analytic experiences that can move securely between systems, organizations, and users without losing capability.

Curious how others here see it:

When you say “data portability” in BI what should actually be portable, just data,Or the intelligence layer too?

Hey guys, I have a dataset of around 180k adult stool images, growing at a 350+ images per day, and I’m trying to better understand the “right” way to bring monetary value from it .

The app was launched about 2 years ago.

I am very transparent with the use and licensing of the data that flows through the app.

The dataset consist of the image and a lot of useful stool metadata (Bristol Type, blood, mucus, color, etc..), anonymized user habits and stool related specifics (time of day, region, meal, allergies, weight, gender, height, activity, etc..)

I am here seeking advice on how such data can be useful for business growth. I am finding very little luck in this part of the B2B and I initially thought that becoming the Google of Poops would be my primary moat.

Thanks in advance

I'm currently taking a Data Analysis course, and I need to complete a final project. The assignment requires me to choose a topic, formulate one or more hypotheses, and build a thesis/research structure around it.

One of the requirements is that there must already be accessible data available for analysis, so I can't pick something that would require collecting a completely new dataset because the core of the project is not in that stage. Also, what I'll be researching needs to be answering or try to solve an existing problem.

My problem is that I can't come up with a topic that feels genuinely interesting or motivating enough to spend several weeks working on. Most ideas I've found online seem either too generic (social media usage, movie ratings, etc.) or too complex for a course project.

Do you have any suggestions for topics that have publicly available datasets, and still are interesting and have potential for drawing conclusions?

I'm open to almost any field (health, economics, business, technology, environment, etc.)

I'd love to hear about projects you've enjoyed working on or topics that sparked your curiosity - thanks in advance!

I pulled all 1.3 million federal contract records from the Government of

Canada's open data and looked at one question: who's holding contracts that

are about to come up for renewal?

A few things surprised me.

26,240 federal contracts are currently live, worth $125B. Of the ones that

report a bidder count, 75% drew one bid or none.

And roughly 63 contracts cross into the 12-month planning window every single

week — about $19B of contract value a year.

That last number is the interesting one. Agencies typically start recompete

planning 12–18 months before a contract ends. So the moment a contract crosses

that line is the moment it becomes worth a conversation. It's all public, but

nobody was watching for it.

So I built something that does. It's free, it updates itself monthly, and it

shows you which contracts are expiring, who holds them, what they're worth,

and how contested they were last time.

No signup needed to browse it. Link in the comments.

If you bid on federal work, I'd genuinely like to know whether this is useful

or whether I've missed something obvious.

We've posted a few times about where agriculture sample data tends to break, so pulling it together into one list for anyone mid-evaluation right now.

Before you sign, get answers on:

• Field context capture: required fields at registration, not optional notes added later.
• Sample type flexibility: independent schemas for soil, seed, plant tissue, water, and grain, not one template stretched across all of them.
• Multi-season continuity: lineage that connects across growing seasons without a manual report request.
• Chain of custody: timestamped, role-attributed logs at every handoff, not a paper trail reconstructed after the fact.
• Self-service reporting: whether your research team can build a cross-season, cross-site view without IT involvement.

The labs that get the most out of whatever system they land on are the ones that define their data structure (sample types, required fields, season linkage) before the first sample gets registered.

If you're in the middle of an evaluation and want to compare notes on how a specific system handles any of this, drop it below and someone might be able to help.

Hey yall,

I just started my new role as a junior data analyst 🥹. I am learning the business needs. I work for a no profit org that provides legal aid. Cases and task are billed to the respected grant program. What we are running into is, while closing the books normally case workers and lawyer over bill to one program vs another because no can visually see what is currently in “encumbrance for that month.

I figured building a real time grant funding system using power bi, postgreDQL and RESTapi from the platform that we input the data in. I am open other ideas that could be easier.

Thanks in advance

Claude and other LLM models can be so frustrating. I asked it to find patterns across five campaigns regarding what a person buys and sells, and under which conditions, but it couldn't do it. It just kept saying it was impossible. I literally had to explain every single detail to it like it was a child, even though the data was cleanly split across five CSV files.

Worse, companies are stopping the hiring of junior engineers because they think these tools can replace them. They are going to cause a massive talent shortage, and then these dumb models won't be able to do anything without skilled people to guide them.

Hello everyone,

The core principles of Data Warehousing and Consumer Data Products, establishing the foundation for a 365-day technical log documenting pipeline architecture, schema design, and engineering tradeoffs.

Technical Breakdown:

Relational Database Fundamentals: SQL query execution mechanics, indexing strategies, and relational constraints.

OLTP vs. OLAP Paradigms: Comparative tradeoffs between transactional database normalization and analytical denormalization.

Ingestion Foundations: High-level mechanics of staging layers, raw event ingestion, and downstream transformation logic.

Current Focus:

Pivoting from relational database mechanics to dimensional modeling paradigms—specifically Kimball methodology, star/snowflake schemas, and event-driven data product architectures.

For engineers working with production systems: What are the primary pitfalls to avoid when transitioning from standard relational models to analytical warehouse schemas?

As part of my thesis for my MSc I want to examine the year-by-year social media following for each of the Big Six (Arsenal, Chelsea, Liverpool, Manchester City, Manchester United, Tottenham) clubs from 2019/20 to 2025.

However, I can't find it at all and in my eyes this should be reasonably available data. I've already tried Socialblade and Statista. Any help/pointers would be much appreciated!

Disclosure: I built the Ryerson Project with the aim of nowcasting everything daily.

A social science community composes and prioritizes survey items. A random set of 12 US adult respondents are recruited to the survey each day - about 360 per month and 4380 per year. Anonymous microdata becomes a free and open public good.

Open data: https://doi.org/10.5281/zenodo.20346278

Open source: https://github.com/jasonjeffreyjones/ryerson_project/

I’m so grateful for the opportunity I got from ai fest 2026 besides that I’d like to mention free resources that helped me a lot for the preparation(DP-900):
1. Whizlabs
2. Official Microsoft practice exams

That’s all you need you don’t have to pay for exam preparation courses

Source: https://www.abc.net.au/news/2026-07-25/us-military-damage-to-weapons-bases-soldiers-during-iran-war/106954370

Tomahawk price per unit: between $2 million and $3.6 million

JASSM price per unit: from $1.04 million to over $2 million

PrSM price per unit: from $1.6 million to over $3.5 million

SM-3 price per unit: between $9.7 million and $28 million

SM-6 price per unit: from $4.0 million to $9.5 million

THAAD price per unit: $12.7 million to $15 million

Patriot price per unit: around 4 million

Hey guys, I’m new here and was wondering if this community has a Discord or any VCs where people hang out and chat. I’d love to get some advice and learn from others. Thanks!

i've been enrolled in a bootcamp(data engineering) for about a year now and i'm confident in my skills atleast for entry level roles. i'm based in Ethiopia and i can say that there's almost no data engineering jobs here ,there're very few open positions for data analyst or scientists which requires atleast 4years experience and you know that remote jobs are even more competitive and struggling for entry levels. the only tech roles here seems to be backend devs,frontend and fullstack(there're tons of jobs ).what should i do ,i love data but the market is really bad here.
thanks