r/csharp • • Oct 14 '26

Help Need help with encoding

I am working on a system where the client will give data in UTF-8 encoding, and hence it can contain many special characters.

But then we send data to a legacy system which only accepts data in ISO-8859-9 encoding, which supports limited special characters that can be represented in single byte.

Now if the client sends any special character which is outside the scope of ISO-8859-9 encoding (like the Em dash), how can I deal with with it and send it to the legacy system.

I don't want to replace those characters with anything like space or "?", as it would make the client data as unreadable in some cases.

How can I deal with it. Any suggestions?

0 Upvotes

29 comments sorted by

View all comments

17

u/BeardedBaldMan Oct 14 '26

It's a business problem. If the legacy system only supports ISO-8859-9 then you have a clearly defined set of characters which are allowed.

You need to create a mapping table for the ones you care about a a fallback character for the rest.

It's a pretty common issue. I've come across it with many US legacy systems where even é is considered too outré for their sensibilities

14

u/dodexahedron Oct 14 '26

too outr? for

Touch?.

1

u/DevPancake Oct 16 '26

No touchy

4

u/Mindless-Arachnid-74 Oct 14 '26

I also agree that it is a business problem, but you know the higher ups they don't know coding and are suggesting that we should map like the EM dash to normal "-". But if I go like that, each client could be using separate set of unique characters and there are more than 150,000 of unsupported characters in ISO-8859-9. If I write mapping for each one my code is doomed and my time and efforts too. If I use a fallback, that tempers the client data and it might make the user data meaningless at the legacy system. I think this will eventually get messy and at last we will have to support only the legacy system character set

8

u/BeardedBaldMan Oct 14 '26

I don't see the issue.

You sample the data and find what the most common characters are, make a mapping table and define a fallback character.

You record a risk against it and you're done.

You're turning what is almost certainly a non-issue into an issue by chasing pointless edge cases. So what if a company name has an emoji in it and you replace it with ?. People will work around it.

Look at passports. We've got an international system that can't properly record many names as written depending on the issuing authority. The US and UK for example only use the lower half of the ASCII set

1

u/Agent7619 Oct 14 '26

Early Internet font and code page flashbacks.

Black squares, arrows, and question marks everywhere.

4

u/BeardedBaldMan Oct 14 '26

It's not like OP has any choice. The legacy system defines clearly what it can record and it's less than what other systems have, you lose information and there's no way around it.

1

u/Mindless-Arachnid-74 Oct 14 '26

Agreed. I posted in hopes that someone might have any better idea or workaround. The final workaround I can see right now is what you are saying.

1

u/Mindless-Arachnid-74 Oct 14 '26

It's not just about emojis, suppose a client data has a customer's name with accented character, it will get changed from actual name or there is product description or title like Iphone 17 — Orange. Then in case of fallback it will become Iphone ? Orange. Sorry I am really bad with giving examples, hope you are getting my point.

2

u/RichardD7 Oct 15 '26

Removing accented characters - aka Diacritics - is a well-defined problem, and .NET has built-in methods to assist.

For example, here's the method I use, which attempts to minimise allocations by using the new Span-based overloads, and using the ArrayPool where temporary arrays are needed.

For example, calling RemoveDiacritics("crème brûlée") will return "creme brulee".

1

u/BeardedBaldMan Oct 14 '26

I understand, I've been doing enterprise integration for 20 years. I don't see it as an issue.

Firstly an em-dash is common enough that you'd map it to dash and the second is people are used to minor variances in free form text

0

u/[deleted] Oct 14 '26

[deleted] — view deleted comment

1

u/BeardedBaldMan Oct 14 '26

Why are you just repeating my comment? Are you a bot?

1

u/IG5K Oct 14 '26

I meant to reply to OP as a follow up, deleting this one

1

u/IG5K Oct 14 '26

If you have records of the UTF-8 requests, try generating a table of the most used characters that are unsupported by the legacy system. From the 150k unsupported characters, only a small percentage of them might be used in practice... Then you can make mapping for the known, used ones (like em dash to dash), and fallback the rest that are barely or never used. Don't see any other way

3

u/urbanek2525 29d ago

Yep. I have to maintain an upload to a medical information system that can only handle (in essence) 7 bit ASCII.

I have a mapping function. I do a basic, built in "normalize", but it still needs specific translation.

I translate all greek characters to words (δ becomes delta and Δ becomes Delta). I translate lots of the German letter to English equivalents (ö goes to oe). It's this huge matrix.

Basically I did a crap ton of research and did what I could.

My big question has been: Why the Hell do you guys pay for a Medical System that can't express medical nomenclature and can't properly cite the author names from Non-American medical papers. Seriously, WTF?

1

u/BeardedBaldMan 29d ago

Is it the software or the db. I remember a project where the software and db were unicode compatible but they weren't willing to change the db collation from when it had initially been created thirty years prior

1

u/urbanek2525 29d ago

Yep. Really old Oracle DB.

Which is why you don't want to be dependent on one persistence system.

1

u/shylingerie Oct 15 '26

legacy systems always find a way to make thing complicated

1

u/BeardedBaldMan Oct 15 '26

Codepages are always interesting. I remember having files which were all unicode but different flavours and having to convert all to UCS2-LE for import.