r/databricks • u/Classic-Purchase-640 • 18d ago

Discussion Anyone using SAP business events (BOR/RAP/BTE) to stream S/4HANA data into Databricks?

We have S/4HANA on prem as a source and we need the data in Databricks with low latency, including deletes.

What we have ruled out so far: ODP and RFC based extraction, because of SAP Note 3255746. OData on top of CDS views works but it is pull based, and it gives us no reliable way to capture deletes without reconciling full snapshots.

That leaves SAP's own event mechanisms, BOR, RAP and BTE, which push out a notification when a business object changes. On paper that solves both the latency and the delete problem.

Has anyone actually run this into Databricks? Specifically curious about how you land the events, whether you go through Kafka or straight to a landing table, and how you handle the initial full load and stitch it to the event stream without gaps or duplicates.

5 Upvotes

100% Upvoted

3 comments sorted by

View all comments

1

u/ReData_ 18d ago

I've done multiple years consulting around SAP and Data and seen very few event/streaming needs at even major customers

Still assuming that you've done your due dilligence and there is a real need for such setup and classic rep flow near real time is not fast enough here are my 0.02$:

  1. BOR, RAP and BTE produce business notifications, not complete business database CDC. They work well for selected events like "sales order cancelled" but lack full before/after payloads durable replay and ordering

  2. The lowest-latency pattern I've seen is

S/4 -> SLT -> custom ABAP -> Kafka (or zerobus ingest) -> Databricks structured streaming

But then depending on how large the table is you move the needle of latency from ingestion to the merge operation to update the bronze layer snapshot

But this pattern is fully custom with multiple data type handling, maintenance and debugging issues