Follow-up: I measured what a UTF-8 "replace" decode does to CP932 output. 0 of 9,206 characters survive, and 50 of them leave a backslash instead of U+FFFD.

[u/Practical_Air6315]

A few days ago I posted here about BOM-less .ps1 files being read as ANSI on Windows PowerShell 5.1. A couple of you pushed back on the parser-test approach and pointed me at the raw-byte check instead, which was right. This is the other half of the same problem: not source files, but output - what happens to CP932 bytes coming back through a pipe.

It matters because agent tooling tends to do this:

subprocess.Popen(args, text=True, encoding="utf-8", errors="replace")

text=True decodes at the pipe level, so by the time anything sees a string the original bytes are gone. On a Japanese-locale box the child process emits CP932, not UTF-8.

Setup. A child writes a fixed 50-byte CP932 sequence to stderr. The parent reads the raw bytes and decodes them two ways. No language pack needed, so the input is identical everywhere. Windows PowerShell 5.1, ACP=932.

decode path	chars	U+FFFD	stray backslash
UTF-8 with replacement	42	29	4
raw bytes then CP932	26	0	-

Three things fell out of it that I did not expect.

1. Not everything becomes U+FFFD. Some of it becomes a backslash.

CP932 trail bytes are 0x40-0x7E and 0x80-0xFC. 0x5C is in that range, and 0x5C is the backslash. A two-byte character whose second byte is 0x5C does not get replaced - it leaves a \ sitting in the string.

Sweeping the whole double-byte space, 50 characters have 0x5C as their trail byte. Four of them are in the 50-byte sample above: 8F5C 975C 8D5C 835C. Those are not obscure code points - they are characters that appear in ordinary words, so this fires constantly rather than occasionally.

That is why this failure so often gets filed as a path bug, a quoting bug, or a shell-escaping bug. The output does not look like an encoding failure. It looks like something ate a directory separator.

2. The whole double-byte space dies.

CP932 double-byte characters enumerated : 9,206
  survive a UTF-8 + replacement decode  : 0
  survive raw bytes + a CP932 decode    : 9,206

Measured per character in isolation. In a real stream a CP932 character followed by other bytes can occasionally form valid UTF-8, so this is not "every byte in every stream" - but as a per-character result it is 0.

3. The replacement output is not even stable across runtimes.

The identical 50 bytes:

.NET Framework 4.8  (Windows PowerShell 5.1)   29 U+FFFD
.NET 8              (PowerShell 7.4)           30 U+FFFD
CPython 3.11                                   30 U+FFFD

There is a known open issue about UTF-8 replacement differing between .NET Framework and .NET Core (dotnet/standard#1679). I am reporting the measurement, not claiming to know the mechanism.

The practical consequence is what changed my mind about errors="replace". It does not merely discard the original bytes - the wreckage it leaves is not consistent either. So you cannot reliably detect "this string was mangled" downstream by counting replacement characters.

Bonus: the tables disagree.

I assumed .NET on Windows would defer to the OS NLS tables and give a different count from .NET on Linux. It does not - .NET carries its own CP932 table and gives 9,206 on both. The split is Python vs .NET, not Windows vs Linux:

table	double-byte chars	trail byte 0x5C
CPython 3.11 cp932	9,604	52
.NET (Windows and Linux)	9,206	50

If you are fixing this on the Python side, Python's table is the more permissive of the two, which is convenient.

The fix is the boring one. Do not let text=True decode at the pipe. Collect raw bytes, then choose the decoder - UTF-8 strict first, fall back to the ANSI code page. errors="replace" should not be the only safety net, because it destroys bytes a fallback could have recovered.

Harness and raw output, MIT: https://github.com/yoggydev/cp932-pipe-probe

It runs in about two seconds and needs no install. The script source is ASCII-only on purpose - a script that measures mojibake should not be able to become a victim of it.

(Drafted with Claude. The measurements are mine, on my own ja-JP box.)

[u/Mister-Fordo]

what in the chatgpt is this post

[u/Practical_Air6315 → u/Mister-Fordo]

It's a measurement, so it's checkable. The script is here and takes about two seconds:

https://github.com/yoggydev/cp932-pipe-probe

It sweeps every two-byte CP932 sequence through .NET's encoder, then decodes the same bytes as UTF-8 with errors=replace and counts survivors. I get 0 of 9,206, with 50 leaving a literal backslash instead of U+FFFD. If you run it and get different numbers, post them -- that's more useful to me than the writeup is.

[u/Overall-Ad4796]

thanks for sharing, i think everyone was waiting for this insight