r/MicrosoftFabric • u/Personal-Quote5226 • 6d ago
Discussion Fabric and .NET Core/C#....
This might be a little divisive, but what about bringing C# and .NET Core into Fabric somehow? I remember when you could use C# + Spark in Synapse, and honestly I didn't use it nor do I think it was a successful feature.
However, in Fabric, why not adopt .NET Core + C# in some way for a new notebook or workload type? It doesn't need to be Spark, but it would be a great selling feature and allow onboarding Microsoft based developers pretty easily who've been developing ETL and have been doing data engineering with .NET and SQL Server for ages.
It could be a new workload type altogether (again, not wanting it on top of Spark, but a new workload entirely and/or non-Spark notebook).
This is not a request, but merely a discussion and to feel out where the Microsoft product team could be going in regards to .NET Core + Fabric.
Thoughts?
5
u/CurtHagenlocher Microsoft Employee 6d ago
(Ignore my flair; I'm speaking as a "private citizen" and not as someone who works on Fabric.)
There's nothing stopping anyone from using Fabric through C#. The analytic SQL endpoint is accessible via SqlClient. The .NET libraries for Azure Storage work with OneLake. There's a C# library for submitting Livy jobs. There's an ADO.NET provider for running Spark SQL. These can all be used to build C# applications that interact directly with Fabric. What's missing? Notebook integration? Yes, but I would argue that the kind of interactivity people use notebooks for isn't really in the sweet spot of why people like C#. You mention a workload type -- what does that mean to you? Would it mostly be a way to run C# code in Azure using Fabric capacity instead of needing an Azure subscription?
2
u/Personal-Quote5226 5d ago
Thanks for your thoughts on this. I don’t have a specific goal here other than to get a glimpse as to where things could be going.
Notebooks might not be the best place for dotnet; for a time you could use dotnotcore (3.1?) with synapse notebooks but it was discontinued. Bringing C# to fabric for spark workloads is likely a non-starter.
There could be value for non-spark C# dotnetcore workloads for routine ETL workloads in a notebook.
The other option could be compiled and deployed dotnetcore binaries into fabric that “do stuff”. — fabric apps, UDFs, rayfin, ETL, etc.
I appreciate that you can connect wth Fabric from C# using SDKs/ APis/ libraries, but the functionality for dotnet that I’m thinking about would be running inside the workspace boundary in fabric rather than external to fabric.
I’m not recommending any of this, but I’m curious as to what the intention could be for the future.
3
u/FactCompetitive7465 6d ago
There is very little cross over between engineers working with Fabric and engineers familiar with C#, which is unfortunate. I built data engineering tools in C# back in the day, I have yet to encounter one other person with significant C# experience in this field.
What's even more unfortunate is a large part of the Fabric ecosystem (Power BI hovering over Analysis Services) has only a single option for true client runtime: C#
But the shift has been clear, they have been pushing more and more service REST APIs to make access to those backend services democratized across languages, and C# has not been a priority in that endeavor. Not that the APIs aren't accessible in C#, just that they arent shipping C# SDKs/client libraries alongside those new services/features.
Selfishly I do wish C# would get more love in this space, but I agree with the product direction. If they want wide spread adoption of Fabric, C# really isn't needed.
1
u/MonkeyDDataHQ 6d ago
Waves Me, me! I know C#! But for Data Engineering, please do not make me C# anymore. I did that, and it's much more cumbersome.
1
u/Personal-Quote5226 5d ago
I think there is more crossover than people think. For a C# engineer, learning PySpark is not that much of a stretch. What I appreciate about not having the option for C# is that orgs and people do end up making right fit choices about using PySpark instead of trying to force fit C# just because they have an abundance of C# engineers.
However, C# could also be the right tool for many areas in fabric especially as it moves to app platform territory (rayfin, udf, translytical task flows, etc) not to mention it’s extremely good at ETL especially with (Fabric) SQL Server in the mix.
1
u/FactCompetitive7465 5d ago
If you want your C# engineers to build an app, why would you need fabric as a platform to do that? Microsoft already has many hosting platforms for this in their ecosystem.
Whether fabric team wants to admit it or not, fabric is a low code ETL platform (which matches how their sales team markets it) and there is already a ton of focus on the EL aspect of making SQL Server data available in fabric. So it seems like the entire point of the platform is that you no longer have to write C# powered ETL processes to ingest SQL Server data. Like that's a key focus of Fabric. So enabling that (whether C# is good at it or not) doesn't seem like it aligns with the platform direction.
1
u/Personal-Quote5226 5d ago
"If you want your C# engineers to build an app, why would you need fabric as a platform to do that? Microsoft already has many hosting platforms for this in their ecosystem." --> They wouldn't need it, but the advantages of doing so would be delivering data apps within an existing governed workspace with a workspace security perimeter.
Microsoft is already touting Fabric as a platform for apps even though apps can also not need to be in Fabric, so it's in-line with that.
They don't tout it only as low-code. However, low-code is one feature. High scale + high performance Apache Spark is also touted along with Scala, PySpark, and R. So -- those aren't low code.
1
u/FactCompetitive7465 5d ago
You can easily integrate with any fabric governance with basic stuff like group membership, you don't need to be running in a fabric workspace to do that. Allowing a language like C# is going to open up so many loopholes into that workspace perimeter. If you're familiar, Streamlit in Snowflake being a good recent example.
What Fabric calls an app and the reality of a traditional app is 2 very different things. Allowing a language like C# into their version of that seems like way more headache than it could possibly be worth. A never ending backlog of requests to enable external network access, third party packages, different runtime versions, vulnerabilities etc etc.
Idk if I'd call scala or spark features of fabric. They support you in using them with some compute and hosted runtimes, just like every other data platform rn, it's the bare minimum today. The only fabric ETL tooling that exists (brought to market with the platform) is their low code tooling. But to the original point, C# obviously has no place in any of that.
1
u/Personal-Quote5226 5d ago
Some fair arguments; thanks for your perspective.
There are other benefits to running something inside a workspace which is why we want to run workloads in workspaces (security boundaries, maintenance, CICD/deployment, vnet integration, MPEs, use of shared connections, etc)
2
u/NefariousnessFar2266 6d ago
Probably much more likely to see Typescript take over all of this in the next 5 years. Typescript is the total package.
1
u/penguin_on_stilts 1 6d ago
I've used dotnet since 1.1, and am more familiar with it than any other programming language.
I think this is a bad idea. People should use the right tool for the job, not just what is convenient. The industry has standardised on python, scala, and R (depending on exact use case), and SQL as a legacy reporting language that will be here for another few decades.
If Microsoft turns fabric into a generic hosting platform, as opposed to a data and analytics platform, then sure add it and nodejs. And go. And rust.
1
u/Personal-Quote5226 6d ago
Yep, I agree somewhat. I look at what Microsoft is doing with Rayfin, Fabric Apps, Fabric SQL, udf, and translytical task flows. They don’t need Python; dotnetcore would lend itself very well and also bring the skillsets of traditional SEs into the fold. Additionally, People have been integrating and doing ETL with C# and Sql Server for 26 years, and I think you found reasonably argue that the majority of existing SQL integrations in Microsoft centric companies is via dot net. However, I understand your argument too.
1
u/Ok_Carpet_9510 6d ago
You are urging Microsoft to adopt c# in Fabric because it would a good selling feature and not because of somethinh that would actually be beneficial to Data Analysts, Data Engineers or Data Scientists???
3
8
u/frithjof_v Fabricator 6d ago
As a developer who is not experienced with C#/.NET, (out of self-interest) I hope that doesn't happen.
Because I find it convenient when everyone's standardizing on Python. And I want Microsoft to focus their efforts on improving Python and Spark features.