After reading about large files and memory problems, I'm suspecting that my code below may be inefficient because it reads the entire files into memory before applying the hash algorithm. Is there a better way??

chunk_size = 1024 hasher = hashlib.md5() while True: try: data = f.read(chunk_size) except IOError, e: log.error('error hashing %s on Agent %s' % (path, agent.name)) return {'error': '%s' % e} if not data: break hasher.update(data) hash_string = hasher.hexdigest() 

After reading about large files and memory problems, I'm suspecting that my code below may be inefficient because it reads the entire files into memory before applying the hash algorithm. Is there a better way?

chunk_size = 1024 hasher = hashlib.md5() while True: try: data = f.read(chunk_size) except IOError, e: log.error('error hashing %s on Agent %s' % (path, agent.name)) return {'error': '%s' % e} if not data: break hasher.update(data) hash_string = hasher.hexdigest()