Powershell generating unknown character when echoing to file

Like Out-File, the default encoding of ">" and ">>" in Powershell 5 is UTF16LE ("Unicode"). I guess psftp can't read that? "FF FE" is the encoding signature or "BOM". How are you running psftp?

format-hex psftp.scp Path: C:\Users\me\psftp.scp 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 00000000 FF FE 23 00 47 00 65 00 6E 00 65 00 72 00 61 00 .þ#.G.e.n.e.r.a. 00000010 74 00 65 00 64 00 0D 00 0A 00 70 00 75 00 74 00 t.e.d.....p.u.t. 00000020 20 00 74 00 65 00 73 00 74 00 2E 00 74 00 78 00 .t.e.s.t...t.x. 00000030 74 00 0D 00 0A 00 71 00 75 00 69 00 74 00 0D 00 t.....q.u.i.t... 00000040 0A 00 .. 

Set the default encoding for ">", ">>" and out-file in powershell 5.1:

$PSDefaultParameterValues = @{'Out-File:Encoding' = 'Ascii'} 

Many years later, an attempt at a systematic overview:

• In the modern, cross-platform PowerShell (Core) edition (v6 and above), it is now fine to use > and >> as they now create BOM-less UTF-8 files (as all file-writing cmdlets now do consistently).

  • Given that many modern programs expect this character encoding, your code will likely work as-is.
  • If you need a different encoding, call a cmdlet with -Encoding, as discussed next.

• In the legacy Windows PowerShell edition (whose latest and final version is v5.1), > and >> default to UTF-16LE (which PowerShell incorrectly calls Unicode), which few external programs understand, and which is the cause of your problem.^[1]

  • > and >> are in effect alias of Out-File and Out-File -Append, and specifying a different encoding requires calling them explicitly with an -Encoding argument.

  • However, with string input, you may use Set-Content and Add-Content, which default to Default, representing the system's active ANSI code page, such as Windows-1252 on US-English systems.

    # `-Encoding Default` implied. # OK with ASCII-only characters, otherwise only if the program # expects ANSI encoding. # If you use -Encoding ASCII, any non-ASCII characters become # *literal ?* chars., i.e. information is *lost*. # (That can also happen for `-Encoding Default` for Unicode chars. # outside the character set of the active ANSI code page.) '#Generated' | Set-Content psftp.scp 'put test.txt' | Add-Content psftp.scp 'quit' | Add-Content psftp.scp 

  • It gets tricky if you want BOM-less UTF-8, because Windows PowerShell's cmdlets do not support creating such files - -Encoding utf8 invariably creates files with a BOM (unlike in PowerShell (Core), where you conversely must use -Encoding utf8bom to explicitly request a BOM). Curiously, New-Item does create BOM-less UTF-8 files, so it can be used to create the file, which allows subsequent Add-Content calls to use -Encoding utf8 safely, because they won't add a BOM to existing file content.

    # New-Item, curiously, creates BOM-less UTF-8 files. # The string must be passed via -Value (or as a *single* pipeline input), # and a trailing newline must be added explicitly. $null = New-Item -Force psftp.scp -Value "#Génerated`n" 'put test.txt' | Add-Content -Encoding utf8 psftp.scp 'quit' | Add-Content -Encoding utf8 psftp.scp 

^{[1] UTF-16LE files created by PowerShell always have a BOM (Byte Order Mark, which is the 2-byte sequence 0xFF, 0xFE, and characters / surrogate-pair halves are represented as 16-bit code units, i.e. as two bytes, with ASCII-range characters having 0x0 (NUL) as the second byte in their two-byte sequence.}

In my case, I was piping a .ps1 into powershell in a .cmd file

type script.ps1 | powershell -ExecutionPolicy Bypass 

Although the script had no errors when run in other ways, when run this way, it would throw a syntax error on the first row (a comment row) due to invalid characters.

I found out that if I saved the .ps1 file with the wScite editor, the default was "UTF-8 with BOM". I changed it to just "UTF-8", and after that piping the script to powershell worked fine.