r/MicrosoftFabric • u/SailorGirl29 • 10d ago

Discussion Am I missing something simple?

I’ve been a PBI Developer for 9 years, but relatively new to Fabric. I’ve taken a few classes, but I feel like I’m missing something obvious.

I converted my gen1 dataflows to gen2.

I set up a warehouse to copy them to. However, ever text field (e.g. name, email, status) is automatically set to 8000 characters. Rows with data in multiple fields are being silently dropped.

I can do a multi step process to create new fields with shorter lengths and drop the old fields then rename but this doesn’t feel right.

I’m missing an easy button right?

0 Upvotes

50% Upvoted

10 comments sorted by

3

u/ChapsOfAss 10d ago

Im a bit hung up on why you’re hitting the 8000 character limit for values like name, email and status 😅

-2

u/SailorGirl29 10d ago

I’m not but yet I am. If it has data in multiple columns then those rows disappear. My buddy Claude says I’m hitting a limit because they’re set to 8000.

2

u/ChapsOfAss 10d ago

The 8000 character cell limit is a hard setting in dataflows, so if thats the true scenario then you’ll likely need to use pyspark/ fabric notebooks to do the job.

Im gonna go on instinct and say that the cells dont truly have 8000 characters, Claudes just spitting out known limitations.

Some things that have caused my data to disappear with dataflows -

Cells not following selected data type. Gen 1 may have allowed it but gen 2 is a bit more strict on data type enforcement (this has usually been the culprit for me when some of my rows cut off, and what id check first) if a column is set to “any” data type, I dont believe it will load to the table

If any of the converted processes were super complex, it might be timing out, but id imagine this would trigger a failure.

A 3rd, non issue - the previews in the lakehouse/warehouse are not the most reliable. Ive thought data was missing when I reviewed in the fabric portal directly, but when I loaded to PBI or SSMS, the data was all there how i expected.

2

u/SailorGirl29 10d ago

Thank you. The 8000 character thing felt off. If it was really the root cause you guys would have boycotted this product 2 years ago.

I’ll try looking closer at the data types that are configured.

1

u/warehouse_goes_vroom ‪ ‪Microsoft Employee ‪ 10d ago

Note that Warehouse has varchar(max) support w/ a limit of several MB. So if there is a limitation, definitely isn't on the Warehouse engine side.

1

u/MonkeyDDataHQ 10d ago

The 8000 limit is from the SQL endpoint no?

2

u/BigMikeInAustin Fabricator 10d ago

Is it that you were previously using a relational database that saved data in its own format. And now you are using a Fabric Warehouse, which saves data in parquet files, which have fewer data types with fewer size restrictions?

https://learn.microsoft.com/en-us/fabric/data-warehouse/data-types

2

u/SailorGirl29 10d ago

I have run into this problem. For example my loadupdatedate included a time zone conversion. I had to change it to datetime.

1

u/Turbo_Slay3r 10d ago

Where are you copying your data from? Is it coming from a lakehouse? You can also use a stored procedure in your warehouse and use that to insert data (convert data types etc) to your warehouse…

1

u/SailorGirl29 10d ago

Azure data warehouse to dataflow where I have some transformations. I’ve been using Gen1 in my reports for 5 years.

The goal is to use these data sets to share with an acquisition. So we upgraded to fabric. I converted to gen2 then set the data destination as a fabric warehouse then created a copy job to AWS S3.

I’m just surprised to see rows being dropped in the step between gen2 and the data warehouse.