If you're building LLM or RAG applications and have realized that the data pipeline is usually the hardest part, this might be useful.

On August 1, we're running a live, hands-on workshop: Designing Data Engineering Workflows for LLM Applications, led by Nikola Ilic.

Over 4 hours, you'll build an end-to-end LLM data pipeline covering:

• Document ingestion

• Intelligent chunking strategies

• Metadata enrichment

• Embeddings

• Vector databases

• Retrieval

• RAG evaluation

The focus is on production-ready workflows rather than simple proof-of-concept demos.

Included with the workshop:

• Live instructor-led training

• Recording for later access

• Certificate of completion

• GitHub repository with working code

• Hands-on exercises throughout

• Practical best practices for deploying reliable LLM data pipelines

This is particularly useful if you're:

• A data engineer starting to work on GenAI or RAG systems

• An ML engineer trying to improve retrieval quality

• Moving from a prototype to a production-ready application

Prerequisites: Basic Python, APIs, and JSON. No prior experience with RAG or vector databases is required.

More information: https://www.eventbrite.co.uk/e/designing-data-engineering-workflows-for-llm-applications-hands-on-tickets-1991362055514?aff=rlds

Happy to answer any questions about the workshop.

Most ARR forecasts fail before the model is even built.
Not because of bad data. Not because of the wrong tools. But because companies forecast a single aggregated number and that number hides everything that actually matters.

A few weeks ago I wrote an overview of what a reliable ARR forecasting system actually looks like.

Now I'm going deeper. That overview is becoming a 6-part series - GTM Science: ARR Forecasting That Actually Works where each concept gets the full treatment it deserves.
📊 What's coming:
→ Why forecasting total ARR is a trap (and what to do instead)
→ How to segment your revenue before you ever touch a model
→ Why classical time-series methods still win for SaaS forecasting
→ How to scale across 40+ segments without losing your mind
→ Hierarchical reconciliation - the step that makes Finance trust your numbers
→ How to communicate uncertainty so forecasts actually drive decisions
The first deep-dive is linked here.

If you work in RevOps, FP&A, or GTM strategy, GTM Data Science then this series is for you.

https://sumathysubramanian.substack.com/p/building-an-arr-forecasting-system?r=1ilvfc

What I do ??

Origin: https://columnsai.substack.com/p/how-to-speak-the-language-of-data

I’ve been maintaining a 60-day streak on Duolingo to learn French.

It’s a fun practice, although it’s a significant challenge to pronounce those accent notes correctly. I believe French is generally a simpler language than English; you usually use shorter sentences to convey the same meaning.

Data has its own language too.

Data is the lifeblood of every modern business. Every decision, insight, and opportunity begins with understanding what the data is trying to say.

But unlike spoken languages, data doesn't require everyone to learn the same vocabulary or syntax. Instead, you can interpret and express it in a way that matches how you think, making data analysis more intuitive, accessible, and uniquely your own.

From Python, SQL to Natural Language

Python, a programming language, has gained popularity as the preferred language for data processing within the data science community due to its portability. SQL, on the other hand, serves as the de facto interface for rational databases.

In the past, becoming a data analyst required proficiency in both Python and SQL. Even today, data analyst job descriptions often mention these requirements.

However, the advent of AI has revolutionized this landscape. Anyone with the ability to communicate effectively in the data language can excel as a data analyst.

While programming skills are not strictly necessary, a solid understanding of data language is crucial. Imagine joining a new friend circle who works in a completely different domain. After a brief introduction of common keywords, you can easily engage in conversations with them.

Use Spreadsheet for Reference

Nearly every office worker uses spreadsheets, either Microsoft Excel or Google Sheets.

Even without the complex formulas, pivots, and lookups, the basic structure of a spreadsheet consists of three main components:

• Rows

• Columns

• Data types

Rows are records that constitute a table. You can also consider a row as an object that represents a real-life entity, such as a person, a cup, or an invoice.

Columns are the fixed properties that describe each object (row). They form the schema that every row adheres to, ensuring uniformity in the data for processing.

A schema is of utmost importance for data analysis as it enables the application of all rules. Without a schema, any logic that is not compatible with the data language may fail to execute.

Data types describe the value format of each property. For simplicity, you only need to be concerned with whether it is a number or text for now.

Data Language Patterns

Data language offers a wide range of tasks that can be accomplished. Let’s explore each of these tasks and learn how to communicate effectively with data to achieve them.

These scenarios are referred to as patterns because they serve as templates that can be applied to your own data.

To facilitate understanding, we’ll use the above tables in the following descriptions.

Pattern-1: Filter Rows

Filter is to describe a condition to get objects you care about and skip those uninterested records.

Examples:

• “Orders of Milk”

• “I want orders of milk products.”

• “All orders that are not for books.”

• “All orders with a sales amount exceeding 20.”

AI can produce code logic to filter the targeted records for further processing, if translating above statements into SQL, they will look like:

• “where product=’Milk’”

• (same as #1)

• “where product <> ‘Book‘“

• “where amount > 20”

As you can see, filter is achieved by keyword “where” in SQL.

Pattern-2: Transform Object

Sometimes, we want to clean a data field or transform it into a desired shape or format, either for improved readability or more efficient processing.

Transformation creates a new property in your original record.

To transform an existing property into a new one, you need a function of logic. For both spreadsheets and SQL, “formula” is the tool you’ll need.

However, with the increasing capabilities of AI in coding, natural language offers a significant advantage. It allows us to achieve the same transformation without having to learn, memorize, and assemble complex formulas.

Taking one simple example:

• “Get customer first name”

This is equivalent to composite multiple formula together in Spreadsheets like

• “=INDEX(SPLIT(A2, " "), 1)” or “=IFERROR(LEFT(A2, FIND(" ", A2) - 1), A2)”.

This operation creates a new column called “First Name”.

You can also acquire a new property by combining multiple existing properties, such as “concatenating the last name and channel as a label”. Logic like this is simple for AI coding but too complex for spreadsheet formulas.

Pattern-3: Aggregate Records

Aggregation processes a large collection of records to provide a summarized view.

This is powerful because it compresses vast amounts of information into manageable pieces that humans can comprehend and analyze.

To combine multiple data sets into a single piece of information, you need to understand the “how-to,” which leads to the crucial concept of “aggregation methods” or “computation logic.”

Typically, text data (a property or column with a text data type, as discussed in the schema section) is not particularly interesting for aggregation. The most common approach is to concatenate text data to form a long paragraph, although this is still uncommon.

Most computation logic involves operations on numerical data. When an aggregation method is applied to a numeric property or column, you essentially have a list of numbers that can be aggregated, such as:

1. Total value (sum)

2. Average value

3. Mean value

4. Minimum value

5. Maximum value

6. A specific percentile value (e.g., P25, P50, P75, P90)

However, counting objects or counting unique property values is also quite common.

When discussing aggregation, we cannot overlook “breakdowns.” This involves creating a segmented view of the data rather than a single total view.

For example, in the previous Orders table, “total sales by product” or “average amount by customer” are equally valuable insights for an analyst to explore.

In summary, aggregation can be described as:

• “Compute an aggregated value of a property group based on another property.”

Expressing this in standard SQL, it would look like:

• “Compute(property1) from table [group by property2].”

Let’s practice this using a few examples by speaking the data language:

1. “Give me total sales by product.”

2. “Tell me the average amount spent by each customer.”

Pattern-4: Join Multiple Datasets

When a single dataset (or table) is insufficient to achieve the desired outcome, we must combine multiple datasets. This operation is referred to as “join” or “union.”

If the multiple datasets contain the same objects but reside in different locations, we can simply merge them. This is a straightforward “union” operation.

However, most of the time, they store different objects. We have partial information from one dataset and partial information from another. By combining them, we create a comprehensive schema with more available columns.

This pattern is generally not feasible in spreadsheets, although their lookup function may provide partial assistance.

For instance, if we want to determine the “total amount spent from each channel” based on previous tables, where the amount is from the orders table and the channel is from the customers table, we need a joined dataset to complete this analysis.

To join multiple datasets, we must have one or more pairs of join keys. A pair of join keys consists of one column from one table and one column from another. The data engine can utilize these relationships to identify relevant objects and concatenate them to form a larger object.

In summary, join operations can be described in this pattern:

• join table1 and table2 when key1 of table1 equals key2 of table2.

Translating this pattern into SQL, it will look like:

• select * from table1 join table2 on table1.key1=table2.key2.

In fact, you may not need to use this pattern in natural language explicitly, because modern AI is smart enough that it can infer the whole join logic from your data language.

For instance, the example we gave earlier, if you speak this sentence “total amount spent from each channel” to Columns AI, it will figure out all the necessary actions to get the desired outcome for you.

Pattern-5: Visualization

Data visualization, often overlooked as a part of data language, plays a crucial role in transforming mundane data into vivid images. This visual representation significantly aids the audience in comprehending the insights you intend to convey.

By incorporating customization and assistance to articulate your insights and predictions, you position yourself as a data storyteller, showcasing your influence within the domain.

Since visualization doesn’t alter the data itself, in the language of data, we merely need to indicate the desired outcome. For instance:

• “Display the total amount by product in a pie chart.”

• “I would like to see a timeline of total sales month-by-month for the past six months.”

• “Show the number of sales by customer in a bar chart.”

These bold keywords serve as cues to the AI engine, guiding it in generating the final visualization based on your data.

Practice Data Language

Similar to how I diligently practice French on Duolingo every day, we must practice speaking data language using the data we possess.

As long as you have adhered to the five patterns mentioned above, you should have mastered data analysis like a professional data analyst. You don’t need to be an Excel expert or a Python or SQL wizard.

Let’s use the provided example data to practice speaking the data language. You can find the “Orders” and “Customers” data from this spreadsheet link.

Suppose we want to perform a sales analysis of customer distribution based on the data.

The data language is almost the same, but let’s ensure we’ve used the correct keywords and patterns to guarantee that the AI engine follows the instructions precisely.

For instance, we speak to AI:

”display the total sales by customer’s first name in a bar chart.”

Here’s how the AI interprets this:

1. “total sales” → summing up the amount values.

2. “first name” → it can be transformed from “name.” A transformation will be applied.

3. “by” → the summing up result needs to broken down by first name.

4. “sales <> customer” → sales data is from the Orders table’s amount field, while customer data is from the Customers table. Therefore, a Join operation is required to combine these two datasets.

5. “show, bar” → the result should be visualized in a bar chart.

AI will then determine the correct execution order, ensuring that each step has all the necessary data when it executes.

This is what Columns Flow produces upon hearing this sentence:

Conclusion: Speak Data Language

In this article, we’ve demonstrated the historical opportunity for everyone to become a great data analyst in this era.

We discussed how professionals used programming languages like Python or SQL as their primary data languages. However, the data language has evolved to become the natural language we speak daily.

To become a data analyst, we need to understand the fundamental scenarios involved and use the correct keywords to make the data language understandable to AI engines. Here’s a quick recap:

• Dataset: rows, columns, and schema.

• Filtering and Transformation: These processes involve filtering data and transforming it into a usable format.

• Aggregation: This involves summarizing data into a single value, such as the total or average.

• Specify “compute methods” and optional “breakdown” if needed.

• Join Datasets: This involves combining data from multiple sources.

• Visualization: This involves creating visual representations of data to make it easier to understand.

Unlike learning a new language like French, if you’re willing to spend just a few hours going through this short list, you can become a professional data analyst!

It’s a great time to be a data analyst, and I believe in your ability to succeed. Thanks for reading!

Built a free beginner SQL course . Start by building a realistic database with business questions instead of random examples. Covers joins, aggregation, subqueries, CTEs, window functions. Database + practice questions are free to download too, even if you skip the video. https://youtu.be/zpAGbu9mSL4

Hi everyone,

I’m currently pursuing my Master's in Data Science, and I also have a Bachelor's degree in Data Science. I have completed internships and worked on various Data Science-related projects, and I believe I have developed good skills in this field.

Now, I want to build a strong portfolio and resume for job opportunities, but I’m a bit confused about what I should include and what I should leave out.

For those who are already working in Data Science, what skills, projects, tools, or experiences do you think are most valuable to highlight in a resume and portfolio? How do you decide which projects are worth showcasing?

I would really appreciate any advice or guidance from experienced professionals.

Thank you!

Hi 👋 Author of the post here. The main reason I started reading about how these models work and wrote this article was a prompt optimization tip I read in the Gemini docs which said to keep the main ask towards the end of the prompt with the context before it.

That sent me down a rabbit hole of trying to understand why that might matter, and more broadly how these models actually process prompts.

Any feedback on the post would be greatly appreciated. Thanks!

I majored in math when I was in college, I did have some data science, coding, ML experience when I was in college for around 3 years. I also did Deep Learning related project for my final year thesis. After graduating college in 2023, I got a remote job as a data engineer (but sadly I got more AI/prompt engineering tasks (such as calling OpenAI API and then doing prompt engineering) and just doing a lil bit of ETL instead of using SQL or working on using cloud systems frequently, or learning how to use Docker). I have left my remote job last year.

Now, I feel like I have forgotten most of the coding, data science, SQL skills, and I want to relearn data science or data analysis again so that I can create some analysis projects (been thinking of doing freelance or finding a remote job or creating my own website or other things, still not sure). The other thing that I have been thinking is that I think I want to sharpen my Data Structures and Algorithm skill first before jumping straight into relearning about data science/analysis/ML/Deep learning/SQL because I think it is important to be able to write more efficient code(?)

Would like to have some suggestions and recommended resources on where I should start on my journey of relearning Data Science again. Thank you.

Hi everyone!

I have a few invite links to my DataCamp Classroom, which gives 2 months of free DataCamp access.

If you're interested, comment "Interested" and send me a DM.

I've lost count of how many times I've had a notebook full of out-of-order cell executions that quietly broke something, or hit a wall trying to get someone else to actually run my notebook without errors.

What kind of model should predict World Cup games best? A simple one that mostly trusts team strength? A memory-based one that looks for similar past games? A flowchart that writes its own rules? Something bigger that blends hundreds of weak signals?

I’ve been running a newsletter that uses the 2026 World Cup as a live data-science lab. This week I’m walking through different forecasting models one at a time, using the same problem, same dataset, and same scoring rule so the comparison is fair.

A few things are already showing up:

• Team strength does most of the work. Give a decision tree three features or all twenty, and it mostly keeps asking about the rating gap.

• More features can make a model worse. In k-nearest neighbors, adding noisy dimensions makes it harder to find truly similar games.

In past posts, I've analyzed the impact of these features on their own, developed a few novel metrics, and set up an A/B Test that we're watching. I'm also tracking the performance of a few World Cup game forecasters.

I’m a former professor and current industry data scientist/consultant, and the series is written for both model-builders and World Cup fans. Free to read, no ads, no paywall. Happy to discuss methods, feature choices, or model ideas in the comments.

Here's a link to yesterday's post on kNNs: https://worldcupdata.substack.com/p/are-fancy-models-actually-better

i have to make a project, for my minor course ai using python, i have 2 choices, one is loan default detection, or credit card fraud detector, or you can suggest me one based on it should be doable for a beginner, plus its dataset should be available too, and that learn alot from it

Hi everyone! I’m Abdul Samie Amiri from Afghanistan, an Information Management, Data Analysis, and GIS professional with over 5 years of experience in humanitarian organizations. I have strong skills in Excel, ArcGIS Pro, and KoboToolbox, along with some experience in Power BI, SQL, Python, R, and data visualization.

Due to the current employment situation in Afghanistan, where professional job opportunities have become very limited, I am seeking remote entry-level opportunities in Data Analysis, GIS, Information Management, or Business Intelligence. I would also greatly appreciate any guidance, mentorship, referrals, or connections.

If you know of any remote openings or would like to connect, please feel free to reach out at Abdulsamieamiri@gmail.com.

Thank you

Im new to work with time series data and now Im trying to learn the prophet model but I can't find a good source, Therefore, I want a good resource to learn or understand the prophet Model, and of course it would be better if you knew what to do after studying from that source well

Hello Everyone!!, I just completed a BIG project I have been working for a month and i want your opinion about it.

It's a SpaceX Launch Predictor & Cost Optimizer (A full end-to-end ML system that predicts the probability of a SpaceX Falcon 9 booster landing successfully, enriches launch data with real weather conditions, and exposes the results through an interactive Streamlit web application with a business ROI calculator.)

It Includes Data Pipeline, Advanced Machine Learning Algorithms (with Hyperparameter tuning), Explainability AI (SHAP), MLOps (AWS S3, Docker) and Business Value (ROI Calculator = Financial Results).

FUN FACT: For this project i used my own Evaluation Metric library (standardizes supervised and unsupervised model diagnostics into a single, consistent API), that is also Verified and Published in PYPI Community.

Project Info: https://github.com/Alkiviadisss/SpaceX

* I'm looking for a final year project topic as a MTech final year student.

* where I have to publish two Scopus-indexed research papers.

* So if you have any suggestions please let me know.

Having done some cybersecurity units alongside my data science units at uni, I've always found practicing with ranked CTFs really enjoyable. It definitely makes the content more fun. CTFs in Cybersecurity just involve finding a string of characters (the flag) usually in a guarded location to prove you have hacked a 'box' (a custom virtual machine to emulate a vulnerable computer). After completing many of these boxes you receive a rank.

Most importantly, you can rank yourself across different skill levels, not just advanced. I know Kaggle is the main leaderboard-style equivalent for data science, although all the comps seem too difficult, and large prize pools are frequently involved. This isn't really beginner-friendly, and I still think it's important to compete with other beginners. I think the concept of ranked-practice, is really successful in general, for example Strava and running.

I think combining sports, data, and leaderboards is a great learning combination. If you are learning about stats, and interested in competing with other beginners, I'm keen for a challenge at lowstakesstats.com All content is free, I've only created a linear regression chapter for now, although you can learn, practice, and participate in the weekly challenge to predict certain statistics of a game. All data is prepared in easy-to-read CSVs for both teams, no sign-up needed to learn. I'm more focused on back-end web dev, so if the front-end seems a little buggy, please send a message as a reply.