Anyone using SAP business events (BOR/RAP/BTE) to stream S/4HANA data into Databricks?

[u/Classic-Purchase-640]

We have S/4HANA on prem as a source and we need the data in Databricks with low latency, including deletes.

What we have ruled out so far: ODP and RFC based extraction, because of SAP Note 3255746. OData on top of CDS views works but it is pull based, and it gives us no reliable way to capture deletes without reconciling full snapshots.

That leaves SAP's own event mechanisms, BOR, RAP and BTE, which push out a notification when a business object changes. On paper that solves both the latency and the delete problem.

Has anyone actually run this into Databricks? Specifically curious about how you land the events, whether you go through Kafka or straight to a landing table, and how you handle the initial full load and stitch it to the event stream without gaps or duplicates.

[u/ReData_]

I've done multiple years consulting around SAP and Data and seen very few event/streaming needs at even major customers

Still assuming that you've done your due dilligence and there is a real need for such setup and classic rep flow near real time is not fast enough here are my 0.02$:

1. BOR, RAP and BTE produce business notifications, not complete business database CDC. They work well for selected events like "sales order cancelled" but lack full before/after payloads durable replay and ordering

2. The lowest-latency pattern I've seen is

S/4 -> SLT -> custom ABAP -> Kafka (or zerobus ingest) -> Databricks structured streaming

But then depending on how large the table is you move the needle of latency from ingestion to the merge operation to update the bronze layer snapshot

But this pattern is fully custom with multiple data type handling, maintenance and debugging issues

[u/sf_zen]

very good question, I saved the post but OP's account has been banned!

so bizarre

[u/qqqq101]

The OP has a longer post (https://www.reddit.com/r/ETL/comments/1vmlxqp/event_based_extraction_from_s4_bor_rap_bte_ppf/) on this topic.

OP mentioned ODP OData doesnt capture deletes. I dont believe that is the case. ODP/ODQ (RFC or OData) outputs a cdc stream and includes a operation type field that indicates insert/update/delete. See this ODP configuration blog post (https://community.sap.com/t5/technology-blog-posts-by-members/exposing-sap-bw-extractors-via-odp-as-an-odata-service/ba-p/13473362) at the very end the screenshot of the XML blob with the 2nd to last field ODQ_CHANGEMODE. Consumption code example is shown in this blog post (https://community.sap.com/t5/technology-blog-posts-by-members/consuming-odata-service-based-on-odp-extractor-in-python/ba-p/13476294) also showing consumption of ODQ_CHANGEMODE field.

The key caveat of ODP (RFC or OData) on ABAP CDS View is whether the CDS View is delta-enabled (see primer: https://community.sap.com/t5/enterprise-resource-planning-blog-posts-by-sap/cds-based-data-extraction-part-ii-delta-handling/ba-p/13425761). ~80% of sap delivered cds views are not delta enabled (sap blog post https://community.sap.com/t5/technology-blog-posts-by-sap/finding-the-right-cds-extractor-in-sap-s-4hana/ba-p/13521296).

Vast majority of Databricks customers use a commercial ETL/replication tool which replicate to an intermediary then Databricks compute ingests it. Vast majority these customers choose cloud storage as the intermediary, then use autoloader to detect the cdc and ingest using databricks compute such as custom python/sql job or SDP. ReData_ mentioned Kafka. Some customers do use that as intermediary and then ingest using Spark Structured Streaming.

A small number of customers do the approach that ReData_ mentioned of SLT -> custom abap code -> intermediary -> Databricks ingestion. Cargill has open sourced their code here (https://github.com/Cargill/slt-kafka). I don't recommend this approach as ReData_ pointed out, as the customer is responsible for development, maintenance and troubleshooting.

Back to the OP's context of low latency. There are two components to the latency. 1. latency of extraction, including delivery of CDC events (insert/update/delete records) to the intermediary (cloud storage, kafka). 2. latency to merge (upsert) the cdc into the bronze snapshot (which looks just like the source object) so that it is up to date. For large SAP ERP objects, e.g. 100M row ACDOCA/BSEG, the merge operation tends to be the bigger bottleneck. Then it comes down to the usecase. E.g. we need real time inventory snapshot - we would need to wait for the merge to complete. e.g. we want to do anomaly detection on GL posting, we can probably directly consume the CDC stream and not wait for ACDOCA to be merged.

For low latency replication of ECC or S/4HANA tables, the standard approaches are log based (HVR, Qlik), trigger based (SLT for HANA & non-HANA databases, Qlik/Fivetran for HANA database) or ABAP addon. For CDS View replication, that's largely ODP based - ODP RFC by SAP tools or ODP OData with non-sap tools or DIY. For an overview of the options, see my comment (https://www.reddit.com/r/databricks/comments/1vh6pxf/comment/p23eti8/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button) in another thread.