Most efficient way to compare a memorystream to a file C# .NET

Another solution:

private static bool CompareMemoryStreams(MemoryStream ms1, MemoryStream ms2) { if (ms1.Length != ms2.Length) return false; ms1.Position = 0; ms2.Position = 0; var msArray1 = ms1.ToArray(); var msArray2 = ms2.ToArray(); return msArray1.SequenceEqual(msArray2); } 

Firstly, getting hashcode of the two streams won't help - to calculate hashcodes, you'd need to read the entire contents and perform some simple calculation while reading.

I'm not sure if I misunderstood it or this is simply isn't true. Here's the example of hash calculation using streams

private static byte[] ComputeHash(Stream data) { using HashAlgorithm algorithm = MD5.Create(); byte[] bytes = algorithm.ComputeHash(data); data.Seek(0, SeekOrigin.Begin); //I'll use this trick so the caller won't end up with the stream in unexpected position return bytes; } 

I've measured this code with benchmark.net and it allocated 384 bytes on 900Mb file. Needless to say how inefficient loading whole file in memory in this case.

However, this is true

It's important to be aware of the (unlikely) possibility of hash collisions. Byte comparison would be necessary to avoid this issue.

So in case hashes don't match you have to perform additional checks in order to be sure that files are 100% different. In such a case following is a great approach.

As you mentioned, this can be done either byte-by-byte or using buffers. Reading data into buffer is a good idea, because it may be more efficient operation when reading from HDD (e.g. reading 1kB buffer).

Recently I had to perform such checks so I'll post results of this exercise as 2 utility methods

private bool AreStreamsEqual(Stream stream, Stream other) { const int bufferSize = 2048; if (other.Length != stream.Length) { return false; } byte[] buffer = new byte[bufferSize]; byte[] otherBuffer = new byte[bufferSize]; while ((_ = stream.Read(buffer, 0, buffer.Length)) > 0) { var _ = other.Read(otherBuffer, 0, otherBuffer.Length); if (!otherBuffer.SequenceEqual(buffer)) { stream.Seek(0, SeekOrigin.Begin); other.Seek(0, SeekOrigin.Begin); return false; } } stream.Seek(0, SeekOrigin.Begin); other.Seek(0, SeekOrigin.Begin); return true; } private bool IsStreamEuqalToByteArray(byte[] contents, Stream stream) { const int bufferSize = 2048; var i = 0; if (contents.Length != stream.Length) { return false; } byte[] buffer = new byte[bufferSize]; int bytesRead; while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0) { var contentsBuffer = contents .Skip(i * bufferSize) .Take(bytesRead) .ToArray(); if (!contentsBuffer.SequenceEqual(buffer)) { stream.Seek(0, SeekOrigin.Begin); return false; } } stream.Seek(0, SeekOrigin.Begin); return true; } 

We've open sourced a library to deal with this at NeoSmart Technologies, because we've had to compare opaque Stream objects for bytewise equality one time too many. It's available on NuGet as StreamCompare and you can read about its advantages over existing approaches in the official release announcement.

Usage is very straightforward:

var stream1 = ...; var stream2 = ...; var scompare = new StreamCompare(); var areEqual = await scompare.AreEqualAsync(stream1, stream2); 

It's written to abstract away as many of the gotchas and performance pitfalls as possible, and contains a number of optimizations to speed up comparisons (and to minimize memory usage). There's also a file comparison wrapper FileCompare included in the package, that can be used to compare two files by path.

StreamCompare is released under the MIT license and runs on .NET Standard 1.3 and above. NuGet packages for .NET Standard 1.3, .NET Standard 2.0, .NET Core 2.2, and .NET Core 3.0 are available. Full documentation is in the README file.

rdfind has an interesting algorithm. Additionally to comparing the size, it looks at the first and last bytes at first (first bytes might often be the same due to standardized file headers). See rdfind