Set custom row delimiter using batch/powershell script

Conceptually simple and memory-efficient, but slow PowerShell solution:

This PowerShell (v2+) solution is slow, but it is conceptually simple and you shouldn't run out of memory, because the input lines are processed one at a time, using #@#@# as the line delimiter.

Note: This solution combines your two steps:

• It replaces the original line breaks with a single space each,

• and it replaces each #@#@# sequence with a newline.

# Create sample input file. @' line 1 starts here and ends here#@#@#line 2 is all on one line#@#@#line 3 spans two lines#@#@# '@ > file # Determine the input file. $inFile = 'file' # Create the output file. $outFile = 'out' $null = New-Item -Type File $outFile Get-Content -Delimiter '#@#@#' $inFile | % { Add-Content -Value ($_.Replace("`r`n", " ").Replace($sep, '')) $outFile } 

Note:

• When you use -Delimiter, the specified delimiter is included in each item passed through the pipeline (unlike the default behavior, where the default delimiter (newline) is stripped).

• Add-Content automatically adds a trailing CRLF to its output (in PSv5+, this could be suppressed with -NoNewLine).

• The approach uses the [string] type's .Replace() method rather than PowerShell's flexible, regex-based -replace operator, because .Replace() performs literal replacements, which are faster (the equivalent command is
  Add-Content -Value (($_ -replace '\r\n', ' ') -replace '#@#@#') $outFile.
  That said, the speed gain is negligible; it's the file I/O part that takes up most of the time).

Much faster PowerShell solution with on-demand compilation of C# code

dbenham's clever and elegant batch + JScript solution is significantly faster than the above PowerShell solution.

Here's an adaptation of his approach to using C# code inside a PowerShell script that is compiled on demand.

Compilation is surprisingly quick (on the order of 0.3 seconds on my late-2012 iMac), and the use of compiled code to process the file results in a significant performance gain.
Also note that compilation is only performed once per session, so subsequent invocations do not pay this penalty.

Processing a ~ 1 GB file (created by repetition of the contents of the above sample file) with the script printed below yields the following:

Compiling... Processing file... Completed: Compilation time: 00:00:00.2343647 File-processing time: 00:00:26.0714467 Total: 00:00:26.3278546 

Execution times in real-world applications will differ based on many factors, but based on @dbenham's timings mentioned in the comments below, the the on-demand compilation solutions is about twice as fast as the batch+JScript solution.

Source code of the fast PowerShell solution:

# Determine the input and output files. $inFile = 'file' $outFile = 'out' # Get current time stamp for measuring duration. $dtStart = [datetimeoffset]::UtcNow # How many characters to read at a time. # !! Make sure that this at least as large as the max. input.line length. $kCHUNK_SIZE = 1000000 Write-Host 'Compiling...' # Note: This statement performs on-demand compilation, but only # on *first* invocation in a given session. $tsCompilation = Measure-Command { Add-Type @" using System; using System.IO; namespace net.same2u.so { public static class Helper { public static void TransformFile(string inFile, string outFile, string sep) { char[] bufChars = new char[$kCHUNK_SIZE]; using (var sw = new StreamWriter(outFile)) using (var sr = new StreamReader(inFile)) { int pos = 0; bool eof = false; string bufStr, rest = String.Empty; while (!(eof = sr.EndOfStream) || rest.Length > 0) { if (eof) { bufStr = rest; } else { int count = sr.Read(bufChars, 0, $kCHUNK_SIZE); bufStr = rest.Length > 0 ? rest + new string(bufChars, 0, count) : new string(bufChars, 0, count); } if (-1 == (pos = bufStr.LastIndexOf(sep))) // should only happen at the very end { sw.Write(bufStr); rest = String.Empty; } else { pos += sep.Length; rest = bufStr.Substring(pos); sw.Write(bufStr.Substring(0, pos).Replace(Environment.NewLine, " ").Replace(sep, Environment.NewLine)); } } } } } } // class Helper "@ if (-not $?) { exit 1 } } Write-Host 'Processing file...' # Make sure the .NET framework sees the same current dir. as PS. [System.IO.Directory]::SetCurrentDirectory($PWD) $tsFileProcessing = Measure-Command { [net.same2u.so.Helper]::TransformFile($inFile, $outFile, '#@#@#') } Write-Host @" Completed: Compilation time: $tsCompilation File-processing time: $tsFileProcessing Total: $([datetimeoffset]::UtcNow - $dtStart) "@