Find the Difference Between Two Rows in Pandas

Finding the difference between two rows in Pandas typically involves calculating how values in columns change from one row to the next i.e the difference in sales data from past month to current month. This task is especially important in time-series data or datasets where temporal progression, rankings, or consecutive comparisons matter. It is done using pandas.DataFrame.diff() in pandas, helps to calculate the difference between two consecutive dataframe elements. 

Method 1: Using the diff() Method

Syntax:

pandas.DataFrame.diff(periods=1, axis=0)

• periods: Represents periods to shift for computing difference, Integer type value. Default value is 1
• axis: Represents difference to be taken over rown or columns. Can take two values {0: rows, 1: columns}. Default value is 0

Example 1: Finding Difference Between Each Previous Row

import pandas as pd data = {'Date': ['2024-11-25', '2024-11-26', '2024-11-27'],'Sales': [200, 250, 300]} df = pd.DataFrame(data) # Calculate difference between rows df['Sales_Diff'] = df['Sales'].diff() print(df) 

Output:

This shows how the diff() function calculates the difference between each row and the one preceding it. The ability to calculate differences between rows in a dataset is essential for identifying trends and patterns.

Example 2: Calculating Difference Over Multiple Periods

You can also calculate the difference over multiple periods by adjusting the periods parameter. Useful when you need to compare data points that are not immediately consecutive, such as comparing sales figures over quarterly or yearly intervals.

import pandas as pd data = { 'Date': ['2024-11-23', '2024-11-24', '2024-11-25', '2024-11-26', '2024-11-27', '2024-11-28', '2024-11-29'], 'Sales': [180, 190, 200, 250, 300, 350, 400] } df = pd.DataFrame(data) # Calculate the difference with a period of 3 df['sales_diff_3_periods'] = df['Sales'].diff(periods=3) print(df) 

 Date Sales sales_diff_3_periods 0 2024-11-23 180 NaN 1 2024-11-24 190 NaN 2 2024-11-25 200 NaN 3 2024-11-26 250 ...

Example 3: Calculating Difference Along Columns in Pandas

The diff() function can also be used to calculate differences along columns by changing the axis parameter. This can be significant in several real-world scenarios, especially in datasets with temporal or categorical columns that represent related measurements. Let's discuss practical use cases with quick examples:

Example : In cases where each column represents a specific time period (e.g., days, months, years), calculating differences along columns helps track changes over time within the same entity. For instance: Comparing sales, revenue, or costs across days, quarters, or years to identify trends.

import pandas as pd data = {'Region': ['North', 'South', 'East', 'West'],'Q1_Sales': [20000, 15000, 18000, 22000],'Q2_Sales': [25000, 18000, 20000, 24000], 'Q3_Sales': [30000, 22000, 25000, 26000], 'Q4_Sales': [35000, 25000, 30000, 30000] } df = pd.DataFrame(data) # Calculate the difference in sales across quarters for each region sales_diff = df.loc[:, 'Q1_Sales':].diff(axis=1) df = pd.concat([df, sales_diff.add_prefix('Diff_')], axis=1) print(df) 

 Region Q1_Sales Q2_Sales ... Diff_Q2_Sales Diff_Q3_Sales Diff_Q4_Sales 0 North 20000 25000 ... 5000.0 5000.0 5000.0 1 South 15000 18000 ... ...

• Regions like "North" show consistent quarterly growth (+5000 per quarter).
• "West" shows smaller growth between Q1 and Q3 but a larger jump in Q4, suggesting an improvement.

Method 2: Using shift() Method

The shift() method shifts data by a specified number of periods, which can then be used to calculate differences manually. Unlike diff(), shift() provides more control over how data is aligned before calculating differences. This is useful for custom calculations like percentage changes.

import pandas as pd data = {'Date': ['2024-11-25', '2024-11-26', '2024-11-27'], 'Sales': [200, 250, 300]} df = pd.DataFrame(data) # Use shift() to calculate differences manually df['Previous_Sales'] = df['Sales'].shift(1) # Shift sales data by 1 period df['Sales_Diff'] = df['Sales'] - df['Previous_Sales'] # Calculate difference print(df) 

 Date Sales Previous_Sales Sales_Diff 0 2024-11-25 200 NaN NaN 1 2024-11-26 250 200.0 50.0 2 2024-11-27 300 250.0 50.0 

shift() gives you control to align data before performing calculations, allowing custom operations like percentage changes or comparisons across non-adjacent rows. Ideal for scenarios where custom conditions are applied during data manipulation.