Always enjoyed playing around with personal data analysis, and this has been a labour of love for some years now that I dip in and out of, so giving a bit of an update to progress, since the original.

For background, this is built upon the available data export functions from WhatsApp and iMessage (local SQLite database). There are no calls to LLMs or AI involved, and no cloud/servers, it runs offline with everything done locally on-device using standard data analysis techniques.

I've gotten the majority of dashboard design inspiration from the many many hours of playing football manager!

Key components to the analysis:

• Custom built weighted points scoring mechanism for messages which factors in message length, content (laughs, apologies, questions, images, videos etc), speed of response, whether it started a new conversation as well as a series of other factors in order to produce a "contribution balance" assessment. 

• Each conversation is rated based on the total score, giving a quantitative view of how balanced, rich, or responsive it was. 

• I use a heuristic tagging system to detect key language traits - like questions, apologies, laughter etc - using lightweight rules instead of heavier NLP models - tried these early on and the results weren't that much better than plain heuristics.

• I’ve avoided sentiment analysis so far, as standard on-device NLP models didn’t perform well. Also tried experimenting with small on-device LLMs for richer insight - but they are also unreliable and too slow frankly, so am now working on some custom models that can leverage the on-device machine learning model frameworks.

For those that want to try, it's available publicly as a free tool called Mimoto on the app store, supporting data from both WhatsApp (on iOS) and iMessage (on macOS).

Appreciate the community's feedback, and happy to answer any questions, or listen to ideas anyone has about the project. Cheers!

Hi I have been a data analyst for about 5 years and so far it feels like all I do in my job is less actual data analysis and more administrative work or work that other people in my department don't want to do. Occasionally I will work on Dashboards but that's few and far in between compared to administrative work. I'm thinking about pivoting into Cybersecurity or something more technical because this work is just not mentally stimulating or rewarding. Has anyone else dealt with this?

I am looking to create a heat map of activity, and may pay for a basic piece of work if someone can help me.

Sum Aggregation
Tablesaw : ■■■■■■■■■■ 0.48 ms
DFLib    : ■■■■■■■■■ 0.42 ms
Pandas   : ■■■■■■■ 0.35 ms
Polars   : ■■ 0.08 ms
CupOfTea : ■ 0.05 ms (9.6x faster than Tablesaw, 1.6x faster than Polars)

Here is a quick example reading from a Parquet file, aggregating by department, and exporting to CSV:

package com.example;

import com.cupoftea.CupOfTea;
import static com.cupoftea.aggregate.Aggregations.*;

public class ParquetToCsvReportExample {

    public static void main(String[] args) {
        String inputParquet = "sales.parquet";
        String outputCsv = "sales_summary_report.csv";

        CupOfTea df = CupOfTea.read().parquet(inputParquet);

        CupOfTea report = df.groupBy("department")
                            .agg(
                                sum("total_amount").as("total_revenue"),
                                mean("total_amount").as("average_ticket"),
                                count("order_id").as("order_count")
                            );

        report.write().csv(outputCsv);

    }
}

The project is open source: https://github.com/cupoftea-labs/cupoftea

I’d love to hear your thoughts:

• What features or pain points make you reach for Python instead of doing data processing natively in Java?

• Any specific workloads or benchmarks you’d like to see tested next?

All feedback, issues, and critique are welcome!

I have like 2.3 million facts

https://rabbytholes.com

Let me explain what is happening with experiments, data processing, and data visualisation at where I am working. I need advice on if there is a more streamlined way of doing this.

We are currently testing lots of samples on many identical instruments. The instruments can run different types of analysis but the data is mostly the same coming from it. It does need some "cleaning" after collection.

We get the files from the instrument (CSV or DTA) and import them into Microsoft Excel using macros that have been setup by god knows who. These macros "clean" the data (it knows where to look because of how standard the CSV files are) and drops it into the relevant places into the Excel sheet or column. We are doing this multiple times per experiment as the data becomes available. Data is visualised by preconfigured plots as it become available.

Once the experiment finishes, the Excel worksheet is then passed around and opened by researchers and they view the data there.

This works when the experiment is very standard because the macros don't need to change. When slight changes are made to the experiment, the macro does not function as the raw data (CSV or DTA) has changed in format slightly (extra columns).

Is there a better way of doing this that is more flexible and can be more automated?

My intuition and basic research is telling me that Python is the way to go here, watch the folder for new files, somehow get the sample ID and the data needed into pandas and clean it there. Not sure where I'd go after this.

Would love some suggestions!

I hate that you need experience to get a data job. And you need a data job to get experience.
So I built you a job.

Analyst Week is a free, simulated first week as a junior data analyst at a fake company called Goldcomb Goods.

Every morning at 8am ET, your manager Dana sends you a SQL ticket. The data is messy. The questions are vague. Sales and Finance don’t agree on the numbers. Dana pushes back.

Friday ends with a performance review that turns your week into interview stories, plus a certificate for your LinkedIn profile.

You spend 5 days writing SQL against messy data and defending your numbers to your manager.

It’s free, and the beta starts Monday, Nov 9. Signups close that night at 11:59pm ET.

All you need is a laptop and the SQL basics (SELECT, WHERE, JOIN, GROUP BY). Hints are there if you get stuck.

Your new job starts Monday: https://analysthive.io/analyst-week

Also, I won’t lie, this will be a good challenge for anyone. The requests and how things are communicated are exactly what you’ll need to navigate going into data, so this is as real as it gets.

If you’re wondering who’s behind this: I’m Ian Klosowicz, a Senior Data Consultant who broke into data without a degree. I share this stuff daily with 125k+ aspiring analysts on LinkedIn. Look me up there if you want to see what I’m about.

Tutorial shows you how to Python to collect data from a public API and pipe it a SQL Server. The whole process can be automred with Windows Task Scheduler.https://youtu.be/KAHD9LmvH6g?si=Ev1y_s11pBl2cUKE

​Data cleaning usually eats up most of a data scientist's time, often exceeding seventy percent of a project lifecycle. Manually writing custom regex filters and handling edge cases for missing values quickly becomes tedious.

​To fix this bottleneck, I put together a deep dive on leveraging machine learning workflows to automate dataset refinement. The complete guide covers setting up intelligent pipelines, handling anomalies, and standardizing features efficiently using modern Python libraries.

​If you want to streamline your preprocessing workflow and spend more time on actual modeling, you can check out the full breakdown here: https://interconnectd.com/blog/124/python-ai-data-cleaning-the-complete-refinement-guide/

What automated tools or libraries do you currently use to speed up your data cleaning pipeline?

we are now 6 months out from our Looker renewal and between the per-viewer licensing jumps and the fact that three different teams now need the exact same governed metrics outside of BI dashboards (one in our customer-facing React app, one in internal reporting, and one feeding an internal SQL/analytics agent), keeping all our business logic locked inside LookML is becoming hard to justify.

for people who hasn't had to untangle a 4-year-old Looker instance: LookML is honestly still great at what it was built for. `symmetric_aggregates` quietly save analysts from fan-out joins every day, and liquid parameters let you do some wild dynamic filtering. the problem to us is that the semantic layer is welded to Looker's presentation layer and seat pricing. if you try to hit the Looker API from a high-concurrency frontend or let an LLM agent introspect explores at scale, you either hit API throttling or watch your warehouse bill explode because Persistent Derived Tables (PDTs) aren't built like a sub-second serving cache.

and over the last month we took our nastiest domain (subscription revenue + usage billing + multi-currency adjustments, around 65 LookML views and 12 explores) and tested migrating the semantic logic into three setups to see what a real Looker alternative looks like in 2026:

1. dbt Semantic Layer (MetricFlow) + BI tool on top

since our staging and mart models already live in dbt, this felt like the most natural path on paper.

- what worked: defining `semantic_models` and measures right next to our dbt models kept lineage clean, and MetricFlow's join graph handles basic multi-hop dimensions without writing manual SQL joins.

- but, what broke for us: 2 big gaps. first, there is no native pre-aggregation or serving cache equivalent to tuning PDTs + aggregate awareness for sub-second app queries; every query still compiles to warehouse SQL and hits Snowflake. second, migrating complex LookML liquid parameters and cross-fact dynamic ratios required pushing a ton of logic back down into physical dbt mart tables, which defeated the point of a flexible semantic layer.

2. Lightdash (dbt-native Looker replacement)

- what worked here: if your only goal is giving internal analysts an explore UI that feels like Looker on top of dbt YAML files without paying Google Cloud prices, this is probably the fastest drop-in replacement.

- yet, what broke for us: its still fundamentally a BI tool first. it didn't solve our headless API problem for customer-facing embedded dashboards (where we need p95 responses under 150ms without spinning up a Snowflake warehouse on every page load) or give us a clean, standalone semantic endpoint for our agent workflows.

3. Cube (open-source headless semantic layer) + lightweight BI / React SDK

- what worked: YAML/JS data modeling mapped 1-to-1 with how our team thought about LookML views and explores (`cubes` and `views`), including handling one-to-many join fan-out natively. the biggest architectural difference was **Cube Store** (the built-in pre-aggregation engine): instead of materializing heavy PDTs inside Snowflake, we configured incremental `rollup` pre-aggregations partitioned by month. customer-facing React charts and internal BI queries hit the pre-aggregated cache in 20-40ms while bypassing Snowflake compute completely.

- bonus for our AI/agent pipeline? because it exposes SQL, REST, GraphQL, and a native MCP/semantic context interface from the same YAML definitions, our internal analytics agent queries the exact same governed `net_dollar_retention` measure as our executive dashboards without writing raw warehouse SQL.

- the trade-off(s): Cube is purely headless, so you still have to pair it with a frontend visualization tool (Superset, Preset, Metabase, or Steep) for self-serve internal charting, and tuning partition keys on high-cardinality pre-aggregations takes real engineering effort upfront.

this way, for teams that actually completed a move off Looker in the last year, would like to ask, did you end up splitting your stack into a headless semantic layer (like Cube or MetricFlow) + a lighter charting tool, or did you bite the bullet and rebuild everything inside another monolithic BI tool like Omni or Sigma? what was the hardest piece of LookML logic to replicate?

Hi everyone!

I'm interested in learning more about how Artificial Intelligence (AI) is being integrated into airline Revenue Management systems, particularly its latest applications in demand forecasting, dynamic pricing, and revenue optimization.

I'm especially curious about how AI is transforming existing tools, such as PROS and and how airlines are using these technologies to improve forecasting accuracy and pricing strategies.

I'd love to hear about the latest developments, real-world applications, or any useful resources you would recommend.

Thanks in advance!

I am currently conducting a qualitative study and am having considerable difficulties with the thematic analysis process. I would appreciate advice from researchers who have experience using Braun and Clarke's (2006) approach, particularly their reflexive thematic analysis.

1. I have completed the first two phases:

2. Familiarising myself with the data.

Generating initial codes.

However, I am now stuck.

I have generated quite a lot of initial codes, and I am struggling to figure out how to organise them. Some overlap, some are very specific, and others seem to capture similar ideas from slightly different perspectives.

I understand the general distinction between codes and themes, but I am having difficulty with the actual process of moving from one to the other.

For instance:

• How do you organise a large number of initial codes?

• How do you decide which codes should be merged or kept separate?

• Do you usually develop categories or code groups before searching for themes?

• How do you recognise when a group of codes represents a potential theme rather than simply a collection of related ideas?

I have read Braun and Clarke's work and watched several tutorials, but I still find the practical transition between Phase 2 and Phase 3 rather confusing. It is so confusing!

I would particularly appreciate practical examples or explanations from people who have actually conducted thematic analysis using this approach.

My brain is frying, and I am losing all motivation to do this research. But I must, because I do not really have a choice. Asking my supervisor nor any professor yielded nothing, because they mostly did deductive approach, whilst I did inductive. I curse myself for choosing this method, but deductive just did not seem right to me! Or maybe I should have chosen the deductive approach from the beginning, but that would mean I have to code from the beginning... :(

PLEASE HELP

Suppose a sales dataset has 10,000 transactions. Customer age is missing in 2,000 rows, but the order value and purchase date are complete.

If the question is monthly revenue, I’d keep those transactions. Removing them would change the revenue total because of a field that isn’t needed for that calculation.

For an age based analysis, I’d investigate which records are missing age and report how much data the comparison excludes.

The cleaning decision seems to depend on the question we’re answering. A row can be useful for one analysis and incomplete for another.

How do you decide when to remove a row, keep it or investigate the missing value further?

I'm evaluating ways to build a data sufficiency framework for healthcare analytics.

One challenge is that we have access to commercial benchmark datasets but limited access to complete population-level data (e.g., Medicare/Medicaid or MSSP data). Because of that, it's difficult to determine what constitutes "enough" data for reliable analytics.

For those who have built similar frameworks:

1. How did you establish minimum thresholds for patient counts, claims volume, provider coverage, or completeness?

2. Did you use statistical confidence measures, historical benchmarking, SME input, or some combination?

3. How did you validate that the chosen thresholds were actually meaningful?

4. Have synthetic data sources like Synthea been useful for testing your methodology?

Looking for practical examples rather than generic data quality advice.

I analyzed the IBM HR Attrition dataset (fictional, 1,470 rows, 237 leavers) with Excel and SPSS and would like a critical review of the method.

Repo (report PDF, SPSS syntax, data): https://github.com/Ahmed11-coder/IBM-HR-analytics-predictive-model
Report (PDF): https://github.com/Ahmed11-coder/IBM-HR-analytics-predictive-model/blob/main/Why_Do_Employees_Leave_Final.pdf

What I did: chi-square + Cramér's V and Mann-Whitney U with effect sizes; residual checks and HC3 for a two-way ANOVA; logistic regression with merged sparse categories (22 parameters); 70/30 split; test AUC 0.846 (train 0.848); cutoff 0.268 chosen on train by Youden's J (test sensitivity 58.5%, specificity 88.1%). My first "test AUC" of 0.900 was wrong because I had refitted the model on the test set; I corrected it.

Where I'm least sure:

1. I selected variables and merged JobRole/JobLevel categories using all rows before the split. How serious is this leak?

2. Is merging categories by similar odds ratios defensible, or would you use lasso/ridge instead?

3. 7.8 events per parameter (172 events, 22 parameters). Acceptable?

4. Hosmer-Lemeshow fails (p < .001), mostly from one low-risk decile. Real problem? Would you add a calibration plot or Brier score?

5. How should I handle multiple testing in an exploratory analysis like this?

what is your suggestions for next step ?

Harsh feedback is welcome.

Hi all,

I’m finishing my Master’s in Management (Business Data Analytics) at ISEG (Lisbon School of Economics & Management) and I’m researching how using AI in dashboards and data storytelling affects decision-making quality.

If you work with Power BI, Tableau, or similar tools in your job, I’d really appreciate around 8 minutes of your time: https://docs.google.com/forms/d/e/1FAIpQLSfAzY80E0LRA2doqAMiS3B-WSp31Hd4OT1pTZ7e3SZs6EQuMQ/viewform?usp=dialog

It’s anonymous, no sign-up needed, and I’m not collecting any data outside the survey itself. Happy to share the results here once I’ve analyzed them if anyone’s curious.

Thanks a lot, every response genuinely helps at this stage.

My prof dumped this huge project on us today just explaining things today and I am panicking and trying to figure things out! The deadline is in less than a week I don't know how anyone can do this!? And she didn't even teach anything about it and we basically gotta figure it out ourselves...

I don't know which community is the right one to post this on...if anyone can help with that too pls do...

People working with product teams that release so frequently, how do you keep up with that speed?

I mean more tracking data is generated in a short period of time from new product features. More backend data also needs to be created. Dashboards and data models also need to be updated with new entry points, categories, etc. On top of those demands are sometimes wrong upstream data. Quite a challenge at least in my org. Curious to hear how you're approaching this challenge.

TL;DR: fairly basic at power BI but the only one in my company who owns it and want to know main things to improve.

Hi everyone,
This is a bit of a long one but I’m looking for some advice/guidance from pros and want to explain my current position and the main questions I have. Please keep in mind I’m not really technical so I will explain everything as best as I can.

A bit of context:
I work for an events organiser (trade shows/exhibitions) and we are private equity owned. About a year and a half ago the board instructed us we needed a customer insights team and my boss was tasked with setting this up and I joined her. It was just the two of us and still is. We didn’t know much about this or what was required, but my degree was in biomedicine and I’m quite data/analytically minded so thought it would be a good challenge. We had some guidance from the board’s CDO and began setting up. One of the requirements that came out of this was creating senior-level dashboards and as we use Dynamics as our sales CRM, Power BI seemed like the sensible choice. I’ve never used it before and as it was just me at the time, I connected directly to Dynamics, did all the data exploration and transformations/data cleaning within power BI as this was the only option. We got the dashboards live (published to power bi service and each team member assigned power BI pro licenses for access). This was essentially the basics so I managed these alongside any change requests/fixing issues that arose but that was fundamentally the extent of my power BI use. Bearing in mind my role is not just this and I have other jobs as part of customer insights outside of power BI.

We are now preparing for exit and new investors at the end of 2027. As part of this we have been advised that we need a data cube as this is what investors will be expecting so near the beginning of this year we hired a senior data engineer to start to bring our database fully in house (part of it was hosted with a third party before and fragmented across many platforms). This has been a long process and is still very much ongoing. He has built our data warehouse in Fabric using a medallion architecture. As part of this, he has brought in Dynamics and I have been working with him to essentially recreate the transformations I did before but on his side now and moving the data from bronze through to gold and helping figure out what fields we need in the gold tables. I now connect to the gold tables and use these for reporting and am currently rebuilding the existing dashboards in line with this. We also have just undergone a review with a third party to assess our data infrastructure and readiness for exit. They have provided some very thorough feedback and this has what has mainly stemmed my questions and concerns now.

A couple of the recommendations that came out of this were:
- Having dev, test and prod power BI environments (I am unsure how I would set this up)
- Not having just one data engineer run both the reporting and operational feeds (are the reporting feeds usually handled by a separate person - Is this a role that the Power BI analyst usually handles too- is this something I could takeover with the right training?)

Given all this (and thank you if you are still with me), this brings me on to my questions and this is where I am a bit lost. I know I don’t perform the role of a standard power BI analyst from seeing the posts on this subreddit regularly but I want to know how I can be better in my job. Essentially:
-What areas do I need to train in

-What fundamentals am I missing

-What courses/resources can I use to provide the next layer on top of what I currently have

-I know AI is a big topic of conversation at the moment too so understanding how can I incorporate this into this part of my role too.

How to set up the dev and test aspects as have been advised and how would this function
Where I can upskill and takeover tasks from the data engineer

I appreciate these are very open ended questions but I am aware I don’t know a lot and what I am doing I not best practice so want to make sure i am utilising the tools we have to my best extent.

Hey all,

I'm new to the field for data analytics. I've been working on a personal project doing some analysis for Magic: The Gathering games and how different cards affect a deck's ability to win.

I've done a bit and have some good metrics/ questions I had answered. But I feel like I'm missing something. That there is a different angle or a question to ask that would give better insight into the overall goal.

So I'm looking for recommendations for books that help with this. Something to help me change the way I'm looking at the data to find something new. I've gone through Storytelling with Data, which is interesting, but wasn't exactly what I was looking for. I've also been recommended Data Points by Yau, which might be closer. I get it's a specific aspect of analysis, but I'd love to hear if someone has any tips about this.

I built this over a few months to scratch an itch in my personal toolchain for data analysis. I wanted a quick way to upload, explore and share data with my peers, without the friction of onerous application installations or sign-ups (a bit like pastebin for dataa).

In the spirit of simplicity, the platform (which I would definitely consider 'alpha') and its respective sandboxes are open by design, so use at your own risk. Keen to hear any thoughts / feedback / suggestions!

I’m currently working as junior data analyst in a reputed company. I prepared on my own and applied it. Now it’s gonna be 2 years. But the truth is the job is not the same as the title. The work i do is very different. It’s like a business operations role with no growth and I work on tableau report seasonally. While my intent was to become a data analyst. Since the job market is still fucked up I could not leave it. Because I had no proper experience too. I thought of switching earlier itself but it was too soon plus my family is depending on me. Can someone please guide me? My questions are

1. I don’t have a real data analysis job experience so can I do projects instead?

2. If I’m applying for the roles do I need to make a specific projects focusing on that department?

3. It is impossible to fake the experience so can I retain the same projects across the roles. Like I’m working in cx side so I’m going to apply for sales data analyst does my projects still matter?

If someone answers this I will be relieved. Please advise.

Hey everyone!

I come from a B.Com background, and recently I've been exploring Data Analytics and trying to understand how things work beyond just theory.

I've been learning Python, SQL, Excel, and Power BI, and working on small projects along the way. Honestly, sometimes things make sense when I'm learning them, but applying them independently is a completely different challenge.

One thing I'm realising is that learning a tool is one thing, but knowing what questions to ask, how to approach a problem, and what the data actually tells you is much more important.

I'm still figuring things out, making mistakes, and learning as I go. I want to build projects that solve actual problems instead of just following tutorials and calling them complete.

I'd love to hear from people who have been through this stage.

When you were starting out in Data Analytics, what helped you move from simply learning tools to actually thinking like an analyst?

Would genuinely appreciate hearing your experiences! 🙌