I am woring with a numpy's 1d array with thousands of uint64 numbers in python 2.7. What is the fastest way to calculate the md5 of every number individually?
Each number has to be converted to string before calling the md5 function. I read in many places that iterating over numpy's arrays and doing stuff in pure python is dead slow. Is there any way to circumvent that?
You can write a wrapper for OpenSSL's MD5() function that accepts NumPy arrays. Our baseline will be a pure Python implementation.
Create a builder
# build.py import cffi ffi = cffi.FFI() header = r""" void md5_array(uint64_t* buffer, int len, unsigned char* out); """ source = r""" #include <stdint.h> #include <openssl/md5.h> void md5_array(uint64_t * buffer, int len, unsigned char * out) { int i = 0; for(i=0; i<len; i++) { MD5((const unsigned char *) &buffer[i], 8, out + i*16); } } """ ffi.set_source("_md5", source, libraries=['ssl']) ffi.cdef(header) if __name__ == "__main__": ffi.compile() and a wrapper
# md5.py import numpy as np import _md5 def md5_array(data): out = np.zeros(data.shape, dtype='|S16') _md5.lib.md5_array( _md5.ffi.from_buffer(data), data.size, _md5.ffi.cast("unsigned char *", _md5.ffi.from_buffer(out)) ) return out and a script compare the two:
# run.py import numpy as np import hashlib import md5 data = np.arange(16, dtype=np.uint64) out = [hashlib.md5(i).digest() for i in data] out2 = md5.md5_array(data) print(data) # [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15] print(out) # [b'}\xea6+?\xac\x8e\x00\x95jIR\xa3\xd4\xf4t', ... , b'w)\r\xf2^\x84\x11w\xbb\xa1\x94\xc1\x8c8XS'] print(out2) # [b'}\xea6+?\xac\x8e\x00\x95jIR\xa3\xd4\xf4t', ... , b'w)\r\xf2^\x84\x11w\xbb\xa1\x94\xc1\x8c8XS'] print(all(out == out2)) # True To compile the bindings and run the script, run
python build.py python run.py For large arrays it's about 15x faster (I am a bit disappointed by that honestly...)
data = np.arange(100000, dtype=np.uint64) %timeit [hashlib.md5(i).digest() for i in data] 169 ms ± 3.14 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) %timeit md5.md5_array(data) 12.1 ms ± 144 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) If you want to put the bindings in a library and compile them at install time, put the following in your setup.py:
setup( ..., setup_requires=["cffi>=1.0.0"], cffi_modules=["package/build.py:ffi"], install_requires=["cffi>=1.0.0"], ) 
_md5.py file in that directory.
cffi version 0.14.5 (released after this was posted if I'm not mistaken), so I would say it is not an old version problem...anyone managed to make fix the 'no lib' problem? Cheers :)_md5.c, _md5.o and _md5.cpython-39-x86_64-linux-gnu.so). Is that what you meant by build the bindings?I would definitely recommend to avoid converting uint64 to strings. You may use struct to get the binary data that can be subsequently fed to the hashlib.md5():
>>> import struct, hashlib >>> a = struct.pack( '<Q', 0x423423423423 ) >>> a '#4B#4B\x00\x00' >>> hashlib.md5( a ).hexdigest() 'de0fc624a1b287881eee581ed83500d1' >>> This would definitely will speed up the process, since there's no conversion, just simple byte copies.
Also, gettig hexdigest() may be replaced with digest(), that returns the binary data, which is faster that converting that to hex string. Depending on how you're planning to use that data later, this might be a good approach.
ATTENTION! Sorry, I missed the question. The code below calculates MD5 of the whole array and without any conversion. This was put into a wrong place.
>>> import hashlib >>> import numpy as np >>> arr = np.array([1, 2, 3, 4, 5], dtype="uint64") >>> m = hashlib.md5(arr.astype("uint8")) >>> m.hexdigest() '7cfdd07889b3295d6a550914ab35e068' 
