different md5sum for the same file content?

Question

I have an issue with computing md5sum. I have a recover tool -which archives file's metadata (inode) and also computes md5sum of them file(s) and stores them in sqlite db during installation. When the file gets removed/deleted . the tool recovers the deleted file using metadata from sqlite-db.It recovers file.Now ,I wanted to make sure recovered file is exactly same as original file.Thus recomputed the recovered files md5sum as shown below. The problem is ,strangely for few files,I can see (using cat) file content are exactly same (as before it was deleted) & stat command shows same output (except different inode number) but md5sum is different.

Following 2 files has same content - thus having different inode number doesn't affect md5sum.

764efa883dda1e11db47671c4a3bbd9e /test/hi1.txt 764efa883dda1e11db47671c4a3bbd9e /test/hi.txt

Any thoughts, how I should proceed with this?

char file_location[512] = {0}; char md5_cmd[512], md5sum[34]; FILE *pf; //some recovery stuff goes here... //Recompute md5 of recovered file memset(md5_cmd, '\0', 512); sprintf(md5_cmd, "md5sum %s", file_location); pf = popen(md5_cmd, "r"); if (!pf) { fprintf(stderr,"Could not open pipe"); return; } // get data fgets(md5sum, 34, pf); if (pclose(pf) != 0) fprintf(stderr, "Error: close Failed."); fprintf(stdout, "Md5sum is %s", md5sum);

,I can see (using cat) What if there are stuff you cannot see ? Control characters, spaces vs tabs, newline at end of one file ? Do a hexdump on the files and compare the hex. — nos
– nos, Commented Nov 11, 2011 at 9:03
Why is it a problem that files with the exact same content has the same MD5? — Jonas Elfström
– Jonas Elfström, Commented Nov 11, 2011 at 9:04
Or just different encoding. MD5 works on the binary representation. — Vladislav Zorov
– Vladislav Zorov, Commented Nov 11, 2011 at 9:06
@nos, thanks thats a good suggestion. I'll try that first.@ Jonas,no the problem is sometimes files with exact same content has different md5sum. @ Thanks,Vladislav, I'll read more about its actual representation. — webminal.org
– webminal.org, Commented Nov 11, 2011 at 9:09
Another (very unpleasant) option is that the system has flaky memory and the md5sum process sometimes hits broken memory locations causing the "verified" data to wiggle around while calculating the sum. These things happen! There is /no/ good reason for an md5sum to change on the same input data. Its purpose is not to change. Vice versa, two different files indeed may have the same md5sum. That's called hash collision and is unlikely but possible and a natural property of hashing functions. — Tilman Vogel
– Tilman Vogel, Commented Nov 11, 2011 at 9:52

glglgl · Accepted Answer · 2011-11-11 09:06:55Z

You cannot reliably compare file contents with cat. This way (unless you use cat -A or such), there can be many difference which go by unnoticed: spaces vs. tabs, whitespace at the end of lines, etc.

You should compare files with

diff -u fileA fileB

or

cmp fileA fileB

.

Thanks guys,using hexdump and diff really shows the file is indeed different,they are (the contents are) not same as i thought.

Collectives™ on Stack Overflow

different md5sum for the same file content?

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related