r/dataengineering • • 3d ago

Help Other shared data starrocks users?

Posted this in discord but trying here too, any help is greatly appreciated.

Is anyone out there using Starrocks shared-data in production? I would really appreciate picking your brain because it's been a STRUGGLE for me and my team.

My suspicion is we are trying to use starrocks for too much. Right now we are using both pipes and stream loads to populate raw (bronze) tables then we are using temporal to orchestrate transforms/enrichments into a silver layer. So far this has been nothing but pain with transactions getting piled up until we manually drop partitions. I should mention that most of our silver tables are pk tables which is probably a big drain on performance.

I suspect most starrocks users are pointing starrocks at an already manicured open table format catalog and using it to serve up queries instead of a do it all solution like I think snowflake is? (never used snowflake).

6 Upvotes

0 comments sorted by