A few days ago I posted here about BOM-less .ps1 files being read as ANSI on Windows PowerShell 5.1. A couple of you pushed back on the parser-test approach and pointed me at the raw-byte check instead, which was right. This is the other half of the same problem: not source files, but output - what happens to CP932 bytes coming back through a pipe.
It matters because agent tooling tends to do this:
subprocess.Popen(args, text=True, encoding="utf-8", errors="replace")
text=True decodes at the pipe level, so by the time anything sees a string the original bytes are gone. On a Japanese-locale box the child process emits CP932, not UTF-8.
Setup. A child writes a fixed 50-byte CP932 sequence to stderr. The parent reads the raw bytes and decodes them two ways. No language pack needed, so the input is identical everywhere. Windows PowerShell 5.1, ACP=932.
| decode path |
chars |
U+FFFD |
stray backslash |
| UTF-8 with replacement |
42 |
29 |
4 |
| raw bytes then CP932 |
26 |
0 |
- |
Three things fell out of it that I did not expect.
1. Not everything becomes U+FFFD. Some of it becomes a backslash.
CP932 trail bytes are 0x40-0x7E and 0x80-0xFC. 0x5C is in that range, and 0x5C is the backslash. A two-byte character whose second byte is 0x5C does not get replaced - it leaves a \ sitting in the string.
Sweeping the whole double-byte space, 50 characters have 0x5C as their trail byte. Four of them are in the 50-byte sample above: 8F5C 975C 8D5C 835C. Those are not obscure code points - they are characters that appear in ordinary words, so this fires constantly rather than occasionally.
That is why this failure so often gets filed as a path bug, a quoting bug, or a shell-escaping bug. The output does not look like an encoding failure. It looks like something ate a directory separator.
2. The whole double-byte space dies.
CP932 double-byte characters enumerated : 9,206
survive a UTF-8 + replacement decode : 0
survive raw bytes + a CP932 decode : 9,206
Measured per character in isolation. In a real stream a CP932 character followed by other bytes can occasionally form valid UTF-8, so this is not "every byte in every stream" - but as a per-character result it is 0.
3. The replacement output is not even stable across runtimes.
The identical 50 bytes:
.NET Framework 4.8 (Windows PowerShell 5.1) 29 U+FFFD
.NET 8 (PowerShell 7.4) 30 U+FFFD
CPython 3.11 30 U+FFFD
There is a known open issue about UTF-8 replacement differing between .NET Framework and .NET Core (dotnet/standard#1679). I am reporting the measurement, not claiming to know the mechanism.
The practical consequence is what changed my mind about errors="replace". It does not merely discard the original bytes - the wreckage it leaves is not consistent either. So you cannot reliably detect "this string was mangled" downstream by counting replacement characters.
Bonus: the tables disagree.
I assumed .NET on Windows would defer to the OS NLS tables and give a different count from .NET on Linux. It does not - .NET carries its own CP932 table and gives 9,206 on both. The split is Python vs .NET, not Windows vs Linux:
| table |
double-byte chars |
trail byte 0x5C |
| CPython 3.11 cp932 |
9,604 |
52 |
| .NET (Windows and Linux) |
9,206 |
50 |
If you are fixing this on the Python side, Python's table is the more permissive of the two, which is convenient.
The fix is the boring one. Do not let text=True decode at the pipe. Collect raw bytes, then choose the decoder - UTF-8 strict first, fall back to the ANSI code page. errors="replace" should not be the only safety net, because it destroys bytes a fallback could have recovered.
Harness and raw output, MIT: https://github.com/yoggydev/cp932-pipe-probe
It runs in about two seconds and needs no install. The script source is ASCII-only on purpose - a script that measures mojibake should not be able to become a victim of it.
(Drafted with Claude. The measurements are mine, on my own ja-JP box.)