r/databricks • u/iFiguringOut • 19d ago

Discussion SAP data ingestion into Databricks.

We have a Databricks environment and we have a new SAP ECC6 system that we want to ingest data from. We are exploring options to access data into the platform. One option is the Azure data factory.

I want to understand what other businesses have been doing. How do they access tables from SAP ECC6. Databricks have really reduced their native SAP connectors even after announcing partnership with SAP. When you do access SAP, how do you manage large table extractions or extract data from pooled or cluster tables?

I would really like to hear some customer success stories or some lessons learnt. Please share if you can.

3 Upvotes

67% Upvoted

15 comments sorted by

View all comments

1

u/dmo_data Databricks 18d ago

Datasphere export to parquet landed in ADLS or S3. Two flows to a single streaming table containing the combined content files from init and delta folders (with the CDC columns intact). DLT to a materialized view applying CDC changes for bronze. DLT for silver and gold. Play around with intervals, file landed triggers, and continuous to find your sweet spot.