Bush hid the facts

File:Windows "Bush hid the facts" bug explained.svg

When a text file is opened in Notepad, Windows checks if the text is encoded in UTF-16 using the Win32 charset detection function {{tt|IsTextUnicode}}. {{tt|IsTextUnicode}} guesses it is Unicode if the total changes to the "low byte" (the even indexes starting at 0) is three times greater than the total changes to the "hi byte" (the odd indexes). If so, it

returns {{tt|true}}, causing the application to incorrectly interpret the text as UTF-16LE.{{cite web |last=Chen |first=Raymond |date=March 24, 2004 |title=Some files come up strange in Notepad |url=https://devblogs.microsoft.com/oldnewthing/20040324-00/?p=40093 |access-date=2022-07-12 |website=The Old New Thing |publisher=Microsoft}} As a result, Notepad renders the text as Chinese characters. It is commonly believed that spaces at even indexes trigger the bug, this is due to space (32) being farther away from the lower-case letters (97...122) than letters are from each other.

The bug had existed since {{tt|IsTextUnicode}} was introduced with {{nowrap|Windows NT 3.5}} in 1994, but was not discovered until early 2004.{{cite web | url = http://weblogs.asp.net/cumpsd/archive/2004/02/27/81098.aspx | title = Notepad bug? Encoding issue? | first = David | last = Cumps | date = February 27, 2004 | work = #region .Net Blog | access-date = February 15, 2009}} Many text editors and tools exhibit this behavior on Windows because they use {{tt|IsTextUnicode}} to determine the encoding of text files. In Windows Vista, Notepad was modified to use a different detection algorithm that does not exhibit the bug, but {{tt|IsTextUnicode}} remains unchanged so any other tools that use it are still affected.{{cite web|url=http://archives.miloush.net/michkap/archive/2008/03/25/8334796.html|title= Bush might've still hid the facts, but he can't hide them from Vista SP1/Server 2008 Notepad!|first=Michael S.|last=Kaplan|date=March 25, 2008 |access-date=13 April 2017}} Modern documentation states "These tests are not foolproof."{{cite web|url=https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-istextunicode|title=IsTextUnicode function (winbase.h)|date=13 Oct 2021|access-date=18 Apr 2025}}

{{Unreferenced section|date=March 2024}}

Several workarounds exist for this bug:

• Add a character so the string is an odd number of bytes long.
• Save the file as "UTF-8" (before 2018) or "UTF-8 with BOM" (after 2018) rather than "ANSI". This prepends a UTF-8 byte order mark which avoids the bug.{{cn|reason=According to ref it is possible to make a string starting with BOM that will fail|date=March 2024}} UTF-8 without the byte order mark would still trigger the bug, as it is identical to the "ANSI" file.
• Saving as "Unicode", which in Microsoft Windows means UTF-16LE. When loading this text {{tt|IsTextUnicode}} should (and does) return {{tt|true}} and the text is correct.
• To retrieve the original text using Notepad, bring up the "Open a file" dialog box, select the file, select "ANSI" or "UTF-8" in the "Encoding" list box, and click Open. Under Windows 2000, Notepad lacks the "Encoding" list box. WordPad appears to load the text correctly without choosing the encoding, since it uses its own encoding detection.

{{Reflist}}

• [https://devblogs.microsoft.com/oldnewthing/20070417-00/?p=27223 The Notepad file encoding problem, redux] – Raymond Chen
• [https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-istextunicode IsTextUnicode] – Microsoft Docs

Category:Character encoding

Category:Software bugs

Category:Microsoft Windows