Hashing a pandas dataframe for calculated column caching

If hashing doesn't work for you can try to take advantage of the internal state of your class.

Cache one method

Use a class attribute as a cache: on first call of the method, store the result into this attribute, and retrieve it on subsequent call.

import pandas as pd class MyClass: def __init__(self, *args, **kwargs): self._df = pd.DataFrame(*args, **kwargs) self._cached_value = None @property def df(self): return self._df @df.setter def df(self, value): self._cached_value = None self._df = value @property def derived(self): if self._cached_value is None: self._cached_value = self._df.sum(axis=1) return self._cached_value cl = MyClass() cl.derived # compute cl.derived # return cached value cl.df = my_new_df_value # cache is emptied cl.derived # compute 

Cache several methods

You can then extend this principle to several methodes using a dict to store the result of each operation. You can use methods name as the keys to this dict (thanks to module inspect, see this response for an example).

import pandas as pd import inspect class MyClass: def __init__(self, *args, **kwargs): self.df = pd.DataFrame(*args, **kwargs) self._cached_values = {} @property def derived(self): method_name = self._get_method_name() if method_name not in self._cached_values: self._cached_value[method_name] = self.df.sum(axis=1) return self._cached_value[method_name] @property def derived_bis(self): method_name = self._get_method_name() if method_name not in self._cached_values: self._cached_value[method_name] = your_expensive_op return self._cached_value[method_name] def _get_method_name(self): return inspect.stack()[1][3] # returns the name of this method's caller cl = MyClass() cl.derived # compute --> self._cached_value = {'derived': your_result} cl.derived # return cached value cl.derived_bis # compute --> self._cached_value = {'derived': your_result, 'derived_bis': your_other_result} cl.derived_bis # return cached value 

You can factorize the bodies of the two properties to respect the DRY principle, but be sure to modify _get_method_name accordingly.

If you know which methods are likely to update your df, you could override them in your custom class, and keep a flag. I'm not going into details here, but here is the basic principle:

import numpy as np import pandas as pd class myclass: def __init__(self, *args, **kwargs): self.df = pd.DataFrame(*args, **kwargs) self.derived_is_calculated = False @property def derived(self): if not self.derived_is_calculated: d = self.df.sum(axis=1) self.derived_is_calculated = True return d def update(self, other, **kwargs): """ Implements the normal update method, and sets a flag to track if df has changed """ old_df = self.df.copy() # Make a copy for comparison pd.DataFrame.update(self.df, other, **kwargs) # Call the base'update' method if not self.df.equals(old_df): # Compare before and after update self.derived_is_calculated = False random_array = np.random.randint(100, size=(2,10)) myobj = myclass(random_array) print(myobj.derived) # Prints the summed df print(myobj.derived) # Prints None myobj.update([1,2,3]) print(myobj.derived) # Prints the new summed df 

There is probably a deeper method of DataFrame or pandas that is called on every change in the DataFrame content, I'll keep looking.

But you could setup a list of methods that your program will use, and make a decorator to do basically what I did in update and call it on each one of the listed methods...

This question is among the Google results for searching how to hash a DataFrame.

For the use case from your example code, caching the result is the best approach, as noted in efont's answer.

To answer the literal question on how to hash a DataFrame and work around the fact that "the hashing function is an expensive step", see this answer by Roko Mijic:

hashlib.sha1(pd.util.hash_pandas_object(df).values).hexdigest() 

Here is the reference for pd.util.hash_pandas_object().