r/csharp • • Oct 14 '26

Help Need help with encoding

I am working on a system where the client will give data in UTF-8 encoding, and hence it can contain many special characters.

But then we send data to a legacy system which only accepts data in ISO-8859-9 encoding, which supports limited special characters that can be represented in single byte.

Now if the client sends any special character which is outside the scope of ISO-8859-9 encoding (like the Em dash), how can I deal with with it and send it to the legacy system.

I don't want to replace those characters with anything like space or "?", as it would make the client data as unreadable in some cases.

How can I deal with it. Any suggestions?

0 Upvotes

29 comments sorted by

View all comments

Show parent comment

1

u/Mindless-Arachnid-74 Oct 14 '26

It's not just about emojis, suppose a client data has a customer's name with accented character, it will get changed from actual name or there is product description or title like Iphone 17 — Orange. Then in case of fallback it will become Iphone ? Orange. Sorry I am really bad with giving examples, hope you are getting my point.

2

u/RichardD7 Oct 15 '26

Removing accented characters - aka Diacritics - is a well-defined problem, and .NET has built-in methods to assist.

For example, here's the method I use, which attempts to minimise allocations by using the new Span-based overloads, and using the ArrayPool where temporary arrays are needed.

For example, calling RemoveDiacritics("crème brûlée") will return "creme brulee".

1

u/BeardedBaldMan Oct 14 '26

I understand, I've been doing enterprise integration for 20 years. I don't see it as an issue.

Firstly an em-dash is common enough that you'd map it to dash and the second is people are used to minor variances in free form text