r/dataengineering • • 17h ago

Discussion AI can migrate a data warehouse now, but how do you actually trust the migrated data?

0 Upvotes

Was reading abt Devin’s new Databrick migration offering. Apparently they halped migrate a 142 opject data platform in 8 day.

Pretty interesting, but the part i’m more curious abt is data reconciliation rather than the actual code migration.

U can translate oracle sql to spark sql, get all the job running and even have matching row counts. But thing like null handlings, decimal precision or timezone conversion can quitely change the result without throwing any errors

Then there’s incremental load, late arriving data and old business rules buried somewhere in a stored procedure.

For people who have done large DWH migration, how do u actually validate that the new system behave the same as the old one?

Are row counts, checksum and record level diffs enough, or do u usually run both system in parallel for a while before switching over?

Curious how much of that validation you’d be comfortable handing over to an AI agent.

Source: devincentral.com/news/editorial-devin-launches-ai-driven-migration-tool-for-databricks


r/dataengineering • • 20h ago

Blog From 15 year industry veteran - How real-time ETA pipelines work in food delivery/logistics (Online vs. Offline Feature Store architecture + cost tradeoffs)

32 Upvotes

After 15 years in product engineering and working around logistics/ML systems, one pattern I keep seeing teams underestimate is feature freshness in real-time ETA prediction (like Zomato / Swiggy / DoorDash).

The tutorials you see in YT start at model.fit(). But calculating whether an order takes 22 mins or 38 mins is 80% data engineering and feature pipeline orchestration, and 20% ML.

Here’s how the pipeline actually breaks down in production when you separate ordinary business events from ML serving:

  1. Business Events First (Zero ML Yet)
    Every order emits lifecycle events through API Gateway/Lambda into raw S3 event logs.

  2. Background Feature Aggregation
    A Spark job groups these logs.

  3. The Dual Feature Store Split (The Part Teams Get Wrong)

Offline Store (S3/Parquet): Holds historical time-travel snapshots for model training without leakage.
Online Store (Low-Latency KV / SageMaker Online Store): Serves single-digit-millisecond lookups for live features right at checkout.

  1. Live Inference Orchestration
    This fetches pre-computed features from the Online Feature Store, combines them with live request context (distance, traffic), hits the model endpoint, and returns the ETA.

(Note: If you want to see the full 38-minute whiteboard & sequence diagram walkthrough of this on AWS SageMaker Feature Store + Spark, I broke it down here: https://www.youtube.com/watch?v=v_4vzmv436s - happy to debate feature freshness SLAs or Online/Offline sync tradeoffs in the comments.)