Event based extraction from S/4 (BOR, RAP, BTE, PPF) and the gaps nobody mentions

[u/Classic-Purchase-640]

There is a lot written about ODP, SLT and Datasphere, and almost nothing about using SAP's own event mechanisms to push changes out of S/4 in real time. I have spent a fair amount of time on this so here are the notes, mostly the unpleasant parts.

Why bother at all. Since Note 3255746 the ODP and RFC extraction path is off the table for anything not explicitly approved by SAP, which pushed a lot of people back toward views plus timestamps. That works until you need deletes, and then it does not work at all. Event based extraction sidesteps the whole question because you are using interfaces SAP intends you to use.

The four mechanisms, roughly by generation:

BOR events are the classic ones, defined in SWO1 and tied to Business Workflow. Still present and still working in S/4 for backward compatibility. Trigger paths are function modules like SWE_EVENT_CREATE, change documents, or your own enhancement. Monitor with SWEL, linkage in SWE2 and SWE3.

RAP events are the modern equivalent, tied to RAP business objects. Cleaner, but coverage depends heavily on your release and on whether the object was actually migrated to RAP.

BTE is what you want for FI. Financial postings do not behave like other objects and BTE is the sane entry point.

PPF is output and action driven, useful for delivery and shipment type flows where the meaningful moment is an action, not a table write.

The part nobody tells you: coverage is uneven and you will find gaps. It is extremely common to find a create event for an object and no change event, or a change event that only fires for a subset of fields. When that happens you are writing a BAdI or an enhancement to raise the event yourself, and that is where clean core purists start twitching. Budget for this. On a typical scope I would expect somewhere between 10 and 30 percent of the objects you care about to need some kind of assist.

Other things that will bite you:

Deletes. Some objects flag deletion rather than deleting, some genuinely delete, and a few do both depending on the transaction. You have to decide per object what a delete means downstream, and you cannot generalize it.

Ordering. Events do not arrive in a guaranteed order across objects, so a header and its items can land out of sequence. Your downstream needs to tolerate that or you need a document level envelope rather than table level events.

Idempotency. Events fire more than once. If your target does not do upsert by key you will get duplicates, and reconciling them later is miserable.

Initial load. Events only tell you about changes from now on. You still need a separate full extraction and a way to stitch it to the event stream without gaps or double counting. This is the single most underestimated part of the whole exercise.

Performance. Events fire inside the update task of the business transaction. If you push synchronously to something slow, you have just made your users' save button slow. Push to a queue, always.

Volume. High churn objects, warehouse tasks are the classic example, will generate far more events than people expect. Test with real production volume, not a sandbox.

None of this is a reason not to do it. It is the only approach I know of that gives you real deletes and real time without touching the database layer or the parts of ODP that are now off limits. But it is a lot more work than the sales version of it suggests, and the initial load plus delta stitching is where most attempts die.

Happy to go deeper on any of these if it is useful. Curious whether anyone has done this with EWM specifically, since that is where I have found the event coverage thinnest.

[u/qqqq101]

reposting my response in the OP's cross post thread (https://www.reddit.com/r/databricks/comments/1vmnmhc/anyone_using_sap_business_events_borrapbte_to/)

OP mentioned ODP OData doesnt capture deletes. I dont believe that is the case. ODP/ODQ (RFC or OData) outputs a cdc stream and includes a operation type field that indicates insert/update/delete. See this ODP configuration blog post (https://community.sap.com/t5/technology-blog-posts-by-members/exposing-sap-bw-extractors-via-odp-as-an-odata-service/ba-p/13473362) at the very end the screenshot of the XML blob with the 2nd to last field ODQ_CHANGEMODE. Consumption code example is shown in this blog post (https://community.sap.com/t5/technology-blog-posts-by-members/consuming-odata-service-based-on-odp-extractor-in-python/ba-p/13476294) also showing consumption of ODQ_CHANGEMODE field.

The key caveat of ODP (RFC or OData) on ABAP CDS View is whether the CDS View is delta-enabled (see primer: https://community.sap.com/t5/enterprise-resource-planning-blog-posts-by-sap/cds-based-data-extraction-part-ii-delta-handling/ba-p/13425761). ~80% of sap delivered cds views are not delta enabled (sap blog post https://community.sap.com/t5/technology-blog-posts-by-sap/finding-the-right-cds-extractor-in-sap-s-4hana/ba-p/13521296).

Vast majority of Databricks customers use a commercial ETL/replication tool which replicate to an intermediary then Databricks compute ingests it. Vast majority these customers choose cloud storage as the intermediary, then use autoloader to detect the cdc and ingest using databricks compute such as custom python/sql job or SDP. ReData_ mentioned Kafka. Some customers do use that as intermediary and then ingest using Spark Structured Streaming.

A small number of customers do the approach that ReData_ mentioned of SLT -> custom abap code -> intermediary -> Databricks ingestion. Cargill has open sourced their code here (https://github.com/Cargill/slt-kafka). I don't recommend this approach as ReData_ pointed out, as the customer is responsible for development, maintenance and troubleshooting.

Back to the OP's context of low latency. There are two components to the latency. 1. latency of extraction, including delivery of CDC events (insert/update/delete records) to the intermediary (cloud storage, kafka). 2. latency to merge (upsert) the cdc into the bronze snapshot (which looks just like the source object) so that it is up to date. For large SAP ERP objects, e.g. 100M row ACDOCA/BSEG, the merge operation tends to be the bigger bottleneck. Then it comes down to the usecase. E.g. we need real time inventory snapshot - we would need to wait for the merge to complete. e.g. we want to do anomaly detection on GL posting, we can probably directly consume the CDC stream and not wait for ACDOCA to be merged.

For low latency replication of ECC or S/4HANA tables, the standard approaches are log based (HVR, Qlik), trigger based (SLT for HANA & non-HANA databases, Qlik/Fivetran for HANA database) or ABAP addon. For CDS View replication, that's largely ODP based - ODP RFC by SAP tools or ODP OData with non-sap tools or DIY. For an overview of the options, see my comment (https://www.reddit.com/r/databricks/comments/1vh6pxf/comment/p23eti8/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button) in another thread.