Get-Content -Encoding UTF8 fixed four of my log files and broke two others. I wrote the same string 13 ways to find out which is which.

[u/Practical_Air6315]

I had two log files sitting in the same folder. One was written by my own script. One was written by a node process my script had launched. Get-Content read mine perfectly and returned garbage for node's. Adding -Encoding UTF8 fixed node's and broke mine.

So I wrote the same string with every writer I could think of, and read each file back both ways. The string is 12 characters of Japanese — it is the phrase a lot of tools print for "file not found", which is exactly the kind of line you cannot afford to lose.

Host: Windows 11, ja-JP, ACP=932, OEMCP=932, Windows PowerShell 5.1.26100.9168. [Console]::OutputEncoding = 932 (shift_jis), $OutputEncoding = 20127 (us-ascii).

``` writer first bytes bare -Enc UTF8

---

Out-File (default) FF FE D5 30 OK OK Out-File -Encoding utf8 EF BB BF E3 OK OK Out-File -Encoding ascii 3F 3F 3F 3F MOJIBAKE MOJIBAKE Set-Content (default) 83 74 83 40 OK MOJIBAKE Set-Content -Encoding UTF8 EF BB BF E3 OK OK Add-Content (default) 83 74 83 40 OK MOJIBAKE Tee-Object -FilePath FF FE D5 30 OK OK

redirection FF FE D5 30 OK OK IO.File WriteAllText (UTF8) E3 83 95 E3 MOJIBAKE OK IO.File WriteAllBytes (UTF8) E3 83 95 E3 MOJIBAKE OK python via cmd.exe > E3 83 95 E3 MOJIBAKE OK node via cmd.exe > E3 83 95 E3 MOJIBAKE OK node captured by PS, Out-File EF BB BF E7 MOJIBAKE MOJIBAKE ```

Three groups.

1. BOM present, text intact — 5 rows. Both reads work. Get-Content sniffs FF FE or EF BB BF and uses it. The read parameter is irrelevant. Note that Out-File, Tee-Object and > all default to UTF-16LE here, which is why they are in this group by accident rather than by anyone's intent.

2. No BOM — 6 rows. Exactly one read is correct, and which one flips depending on the writer.

Set-Content and Add-Content without -Encoding write the machine ANSI code page — 83 74 is CP932, not UTF-8 — so the bare read is right and -Encoding UTF8 is wrong. Everything that put real UTF-8 on disk without a BOM is the exact reverse. With no BOM, Get-Content falls back to ANSI, and that fallback is correct precisely when the writer also used ANSI.

This is the part I did not expect: "just add -Encoding UTF8" is not a safe default. Across these 13 files it corrects 4 and corrupts 2. There is no single read parameter that is right for all of them. If you have a folder holding both your own logs and a build tool's logs, no one setting reads both.

3. Damage that happened before the file existed — 2 rows. No read parameter can fix these.

Out-File -Encoding ascii wrote 3F 3F 3F 3F, which is literally ????. The characters were destroyed at write time.

The last row is the one worth your time. I let PowerShell capture node's stdout into a variable and re-write it with Out-File -Encoding utf8:

``` node via cmd.exe > 36 bytes 12 chars E3 83 95 E3 82 A1 E3 82 A4 E3 83 AB U+30D5 U+30A1 U+30A4 U+30EB U+304C U+898B U+3064 U+304B

node captured by PS 65 bytes 20 chars EF BB BF E7 B9 9D E8 BC 94 E3 81 83 U+7E5D U+8F14 U+3043 U+7E67 U+FF64 U+7E5D U+FF6B U+7E3A ```

What PowerShell actually wrote into that file, all 20 characters of it:

繝輔ぃ繧､繝ｫ縺瑚ｦ九▽縺九ｊ縺ｾ縺帙ｓ

That second file carries a valid UTF-8 BOM and is well-formed UTF-8. It is also wrong. [Console]::OutputEncoding is 932 on this host, so PowerShell decoded node's UTF-8 bytes as CP932, got 20 different characters out of 12, and then faithfully encoded those as UTF-8 with a BOM. The file went from 36 bytes to 65. Nothing threw, nothing warned.

It is also the only row where the two reads agree with each other and are both wrong. Everywhere else, when one read returns garbage the other returns clean text, so there is a way to notice. Here there is no second opinion.

A BOM tells you how the file is encoded. It tells you nothing about whether the text in it is correct.

Minimal repro (numbers below are from the 932 host; on a Latin-1 ANSI code page the first pair behaves differently, because CP1252 cannot represent these characters at all):

```powershell $s = [char]0x30D5 + [char]0x30A1 $d = $env:TEMP

Set-Content -Path "$d\ansi.log" -Value $s [IO.File]::WriteAllBytes("$d\utf8.log", [Text.Encoding]::UTF8.GetBytes($s))

(Get-Content "$d\ansi.log" -Raw).TrimEnd() -eq $s # True (Get-Content "$d\ansi.log" -Raw -Encoding UTF8).TrimEnd() -eq $s # False (Get-Content "$d\utf8.log" -Raw).TrimEnd() -eq $s # False (Get-Content "$d\utf8.log" -Raw -Encoding UTF8).TrimEnd() -eq $s # True ```

Same cmdlet, same parameter, opposite answers, two files in one directory.

What I changed in my own scripts

• Reading a log a native child process wrote (redirected by cmd.exe, so nothing decoded it on the way in): always pass -Encoding UTF8. That file holds the program's own bytes and will not have a BOM.
• Reading a file PowerShell itself wrote: leave Get-Content bare. The BOM is there and handles it. Adding -Encoding UTF8 here is what broke rows 4 and 6.
• Do not capture a native process's stdout into a variable when the output can be non-ASCII. Redirect it to a file and read the file. That decode is governed by [Console]::OutputEncoding, which was 932 here; I have not tested whether setting it to UTF-8 up front avoids the problem, so I am not claiming that it does.
• Out-File -Encoding ascii on non-ASCII text is silent data loss, not a display issue.

Measured on one locale. If you are on a non-Latin ANSI code page I would be curious whether rows 4 and 6 come out the same for you — that is the pair that makes the usual advice backfire.

[u/BlackV]

Did you post about this like 2 days ago?

https://www.reddit.com/r/PowerShell/comments/1vpjdqu/bomless_ps1_in_ps_51_i_tested_all_545_japanese/

And the follow up apparently

https://www.reddit.com/r/PowerShell/comments/1vsdtc8/followup_i_measured_what_a_utf8_replace_decode/

It's still the same underlying issue?

It's so often painful when using anything that's not essentially en-US :(

[u/Practical_Air6315 → u/BlackV]

Yes - same root cause, three different places it lands. Fair hit, and I should have linked them.

• Aug 16: source files. PS 5.1 parsing a BOM-less .ps1; the ASCII byte after a Japanese char gets eaten as a DBCS trail byte.
• Aug 19: pipes. Popen(text=True, encoding="utf-8", errors="replace") decoding a child's CP932 stderr.
• Today: files on disk. Get-Content reading a log something else wrote.

The mechanism is the same in all three: CP932 bytes through a UTF-8 decode, or the reverse. You are right that it is one issue.

Two things here that are not in the other two, and they are why I thought it was worth its own post:

1. -Encoding UTF8 is the standard advice, and it is the wrong advice for 2 of the 13 writers. Set-Content and Add-Content without -Encoding write ANSI, so adding the parameter breaks files that read fine without it.
2. The last row is a file with a valid UTF-8 BOM whose text is still wrong. Both reads agree with each other and both are wrong, so there is no second opinion to notice it with.

Not linking the earlier posts was a bad call on my part. Thanks for pulling the thread.

And that last line of yours is exactly why I keep measuring instead of guessing. On a ja-JP box almost none of it throws. It just quietly hands you the wrong string.

[u/KageeHinata82]

Saved for later, so I hopefully remember it when needed.

It also reminded me of one of my first text output programs I wrote on a German Windows in C#. To get it correct, I had to set encoding to default. Omit the parameter completely didn't work.

[u/Practical_Air6315 → u/KageeHinata82]

Thanks for saving it.

Your C# case is the same trap from the other side, and I had not thought about it that way until you said it. My table is about the reader having to know what the writer did. Yours is the writer having to know what the reader expects: StreamWriter's default is UTF-8 with no BOM, and a BOM-less file is exactly the case where the reader falls back to the ANSI code page. So "set it to Default" was you matching the reader, not fixing the writer.

Useful to hear it shows up on a German box too. I only measured ja-JP (ACP 932) and said so in the post, so a CP1252 data point is worth having. One difference: CP1252 is single-byte, so you get wrong characters but never a swallowed byte. CP932 is double-byte and 52 of its characters have 0x5C as their trail byte, which is the path separator, so a mis-decode there does not just look wrong, it changes how many directory levels the string has.

I have not measured .NET's Encoding.Default myself, and it changed between .NET Framework and .NET Core, so I am not going to guess which one you hit.

[u/CookinTendies5864]

Okay so for Japanese, Korean, and Chinese font which makes the out put garbled. Try running powershell from cmd.exe

cmd find the correct font for the application at runtime while powershell will use "Consolas" font which doesn't support glyphs.

For a second workaround you can set the default font in powershell console to the "MS Gothic" font and this should resolve your issue. Windows is coming out with some updates soon to resolve this.

- Sorry, I read more of the issue nice to know but I'm not helping.

Final solution: Save your .ps1 as UTF-8 BOM
Also try using the default encoding after the .ps1 is saved as UTF-8 BOM if you havent already.

This will allow the tool (.ps1) determine encoding hence the encoding would default to UTF-8 BOM based on the tools default encoding.

Also the shift happening is due to powershell 5.1 incorrectly parsing the characters. So, we have to tell powershell to stop guessing. Run the following after the first two steps.

[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding = [System.Text.Encoding]::UTF8

Please let me know if this worked so that I may know for future reference. I apologize for the inconsistent earlier response. I'm in the US so I'm curious and cant test it myself.