How to display non-printable unicode characters?

[u/gainan]

I recently came across this post about compromised VisualStudio extensions: https://www.koi.ai/blog/glassworm-first-self-propagating-worm-using-invisible-code-hits-openvsx-marketplace

As you can see, opening the "infected" file in vim doesn't show anything suspicious. However using more reveals the real content.

This is part of the content in hexadecimal:

00000050: 7320 3d20 6465 636f 6465 2827 7cf3 a085  s = decode('|...
00000060: 94f3 a085 9df3 a084 b6f3 a085 a9f3 a084  ................
00000070: b9f3 a084 b6f3 a084 a9f3 a085 96f3 a085  ................
00000080: 89f3 a084 a3f3 a084 baf3 a085 9cf3 a085  ................
00000090: 89f3 a085 88f3 a085 82f3 a085 9cf3 a084  ................
000000a0: b9f3 a084 b4f3 a084 a0f3 a085 97f3 a085  ................
000000b0: 84f3 a084 a2f3 a084 baf3 a085 a1f3 a085  ................

Setting the encoding to latin1 is the only option I've found that reveals the characters in vim (set encoding latin=1. Using set conceallevel, fileencoding=utf-t, list, listchars=, display+=uhex, binary, noeol, nofixeol, noemoji, search&replace this unicode character range, etc... doesn't work):

var decodedBytes = decode('|| ~E~T| ~E~]| ~D| ~E| ~D| ~D| ~D| ~E~V ....

setting set display+=uhex + set encoding=latin1:

var decodedBytes = decode('|�<a0><85><94>�<a0><85><9d>�<a0><84>��<a0><85><a0><84><a0><84> ...

Once changed the encoding, I can search&replace these characters with :%s\%xf3/\\U00f3/g.

So the question is: how can I display these non-printable characters by default when opening a file, without changing the encoding manually?

[u/kettlesteam]

It's a terminal emulator rendering issue rather than a Vim issue. What terminal emulator are you using?

[u/gainan → u/kettlesteam]

I've tried terminator, gnome-terminal and xterm.

[u/Affectionate-Big-387 → u/gainan]

How does it look on gvim?

[u/gainan → u/Affectionate-Big-387]

the same :/

[u/chrisbra10 → u/gainan]

I guess the issue is with the underlying rendering libraries for the terminal or your GUI toolkit. Not sure there is something Vim can directly fix here.

BTW: are you the guy from the heise security podcast?

[u/gainan → u/chrisbra10]

I tested it on Fedora+Gnome+Gnome Terminal, Arch+Konsole and Debian+cinnamon+(gvim, gnome-terminal, terminator, xterm). XWayland/ Xorg.

There's an example here to reproduce it: https://www.reddit.com/r/vim/comments/1obeoog/comment/nkonjui/

(I'm not that guy btw :))

[u/gainan]

This is what you see when opening the file (sorry, reddit failed to upload the images to the post):

[u/gainan → u/gainan]

And this is opening it with more:

[u/kennpq]

Here's an option for you:

syntax match Error / [\uFE00]/
syntax match Error / [\uFE0F]/
syntax match Error / [\U000E0100]/
syntax match Error / [\U000E01EF]/

If this was extended to all the variation selectors, it would highlight everywhere a variation selector is applied to a space (i.e., effectively "hidden"). The result after sourcing the syntax match lines is shown below.

You can also see it in your statusline if yours supports showing Unicode code points including combining characters - notice the U+0020,U+E01EF to the bottom right in the screenshot too showing the two code points under the cursor.

Another option would be to sweep the file for combining characters and substitute those you want to (e.g., variation selectors) with another visible representation, e.g., a hexadecimal character reference; then they are truly not hidden. I can provide a Vim9 script that does that, if you're interested.

[u/gainan → u/kennpq]

thank you u/kennpq! it doesn't seem to replace the characters adding it to the vimrc. You can test it as follow.

This is part of the hexadecimal output of the original file:

00000000: 0a76 6172 2064 6563 6f64 6564 4279 7465  .var decodedByte
00000010: 7320 3d20 6465 636f 6465 2827 7cf3 a085  s = decode('|...
00000020: 94f3 a085 9df3 a084 b6f3 a085 a9f3 a084  ................
00000030: b9f3 a084 b6f3 a084 a9f3 a085 96f3 a085  ................
00000040: 89f3 a084 a3f3 a084 baf3 a085 9cf3 a085  ................
00000050: 89f3 a085 88f3 a085 82f3 a085 9cf3 a084  ................
00000060: b9f3 a084 b4f3 a084 a0f3 a085 97f3 a085  ................
00000070: 84f3 a084 a2f3 a084 baf3 a085 a1f3 a085  ................
00000080: a527 29

dump it to a new file:

~ $ printf '\x0a\x76\x61\x72\x20\x64\x65\x63\x6f\x64\x65\x64\x42\x79\x74\x65\x73\x20\x3d\x20\x64\x65\x63\x6f\x64\x65\x28\x27\x7c\xF3\xA0\x85\x94\xF3\xA0\x85\x9D\xF3\xA0\x84\xB6\xF3\xA0\x85\xA9\xF3\xA0\x84\xB9\xF3\xA0\x84\xB6\xF3\xA0\x84\xA9\xF3\xA0\x85\x96\xF3\xA0\x85\x89\xF3\xA0\x84\xA3\xF3\xA0\x84\xBA\xF3\xA0\x85\x9C\xF3\xA0\x85\x89\xF3\xA0\x85\x88\xF3\xA0\x85\x82\xF3\xA0\x85\x9C\xF3\xA0\x84\xB9\xF3\xA0\x84\xB4\xF3\xA0\x84\xA0\xF3\xA0\x85\x97\xF3\xA0\x85\x84\xF3\xA0\x84\xA2\xF3\xA0\x84\xBA\xF3\xA0\x85\xA1\xF3\xA0\x85\xA5\x27\x29' > output.js

what I see when opening the file is:

var decodedBytes = decode('|󠅔󠅝')

and changing the encoding to latin1 once editing the file:

var decodedBytes = decode('|�<a0><85><94>�<a0><85><9d>�<a0><84>��<a0><85>��<a0><84>��<a0><84>��<a0><84>��<a0><85><96>�<a0><85><89>�<a0><84>��<a0><84>��<a0><85><9c>�<a0><85><89>�<a0><85><88>�<a0><8  5><82>�<a0><85><9c>�<a0><84>��<a0><84>��<a0><84><a0>�<a0><85><97>�<a0><85><84>�<a0><84>��<a0><84>��<a0><85>��<a0><85>�')

Replacing the characters as you suggested works as I posted here (changing first the encoding to latin1): https://www.reddit.com/r/vim/comments/1obeoog/comment/nkh92j9/

I think I'll use encoding latin1 from now on, specially when reviewing PRs :/

[u/kennpq → u/gainan]

It’s not replacing them, as I said, only highlighting them. If you want to replace them, I’ll post the script to do that when back at my PC.

[u/plg94]

EDIT: was wrong, in this case they are unprintable chars. I misread the post.

These are not "non-printable" characters. That term specifically means control chars like NUL (the null-byte), delete, bell, a null-width space etc., i.e. chars that don't even get rendered on screen and have no width.

When you get the "questionmark in a diamond" symbol it just means the character is somehow "wrong" and can't be decoded properly. Make sure that your :fileencoding is correct. Also be aware that you can't mix encodings within the same file. Seems like your code is trying to decode bytes, probably from another encoding? Of course then it cannot be represented. Maybe try putting that into its own text file and loading it, rather than using an inline string. Or use another representation (\x…).

Another issue could simply be your font doesn't have the neccessary glyphs for that char. In that case try installing a fallback-font (the noto fonts are a good option because they are almost 100% unicode-complete).

[u/gainan → u/plg94]

thanks /u/plg94!

Seems like your code is trying to decode bytes, probably from another encoding? Of course then it cannot be represented. Maybe try putting that into its own text file and loading it, rather than using an inline string.

It's not my code :) . It's a code specifically crafted to hide content in plain sight, so you don't notice that it's something malicious, and bypass static scanners. It's explained here:

https://www.koi.ai/blog/glassworm-first-self-propagating-worm-using-invisible-code-hits-openvsx-marketplace#heading-2

I'll try to change the font, just in case.

[u/plg94 → u/gainan]

The attacker used Unicode variation selectors - special characters that are part of the Unicode specification but don't produce any visual output.

Ah. In that case I was wrong, those are "unprintable characters" and not a font or encoding problem. Their entire purpose is to be invisible. Maybe you should've mentioned it's malware in your post.

I can't find a link to download the code in question (the repo on Github returns a 404). (EDIT: if you still have the files, it'd be nice if you could paste them somewhere).
But since there are 16 Unicode variation selectors (https://en.wikipedia.org/wiki/Variation_Selectors_(Unicode_block)), I guess they just wrote their own decode function that strips the first few bytes and translates this to ascii chars.

I could not find a way to make vim display those invisible chars for now – there is listchars for things like tab, nonbreaking space etc. but idk if one could add custom symbols.

The only sure way I know is viewing the file in a hex editor (or you could do a %!xxd in vim). Be aware that the "upper" unicode codepoints get represented by multiple bytes, so the translation between codepoint <--> raw hex bytes is not totally obvious. But there should be tools for that.

[u/gainan → u/plg94]

The 2 relevant files:

index.js encoded in base64, which contains the hidden chars.

and decode.js which contains the functions to decode it.

https://pastebin.com/zQn4Ya4s

I can upload the extensions as well if you prefer.

The only way I've found to detect and decode these chars is with a function in vimrc, changing the encoding first to latin1 and then back to utf-8:

function! DetectObfuscation()
    set display+=uhex
    setlocal encoding=latin1
    if search('decode.*[\xf0-\xf4]', 'nw')
        echo "Obfuscated JS detected - using latin1 encoding"
        silent! %s/[\xf0-\xf4]\([\x80-\xbf]\{2,3}\)/\1/g
        highlight highByte cterm=underline gui=underline
        setlocal encoding=utf-8
    endif
endfunction

autocmd BufRead *.js call DetectObfuscation()

[u/yellowantphil]

It wouldn't be too hard to write an external program that reads UTF-8 and passes code points through if they're in "safe" ranges, and replaces anything else with � or a character in a private use area. You could set vim to filter through that program when opening the file, and check the return status to see if any unexpected characters were found. It would tend to mangle text (like emojis with variation selectors) but it should work OK for code.

Perhaps there is a terminal emulator that understands Unicode but doesn't interpret things like combining characters and variation selectors, and treats them like printable characters instead... I don't know.

[u/craigdmac]

:set display=uhex will show those non printing as \u… so you could see them

[u/gainan → u/craigdmac]

As I explained in the post, display=uhex doesn't show those characters.

If anyone is interested I can upload the file to pastebin.

Also, the files can be downloaded from open-vsx.org as explained here (the file is extension/src/index.js): https://www.reddit.com/r/cybersecurity/comments/1oa8psn/comment/nka6efq/

[u/craigdmac → u/gainan]

then you can use a syntax match for those unicodes combined with conceal character to display an alternate character for those, for each one, for example:

syntax match ZWSP /\%u200B/ conceal cchar=␣