How can I normalize the data in a range of columns in my pandas dataframe

Question

Suppose I have a pandas data frame surveyData:

I want to normalize the data in each column by performing:

surveyData_norm = (surveyData - surveyData.mean()) / (surveyData.max() - surveyData.min())

This would work fine if my data table only contained the columns I wanted to normalize. However, I have some columns containing string data preceding like:

Name State Gender Age Income Height Sam CA M 13 10000 70 Bob AZ M 21 25000 55 Tom FL M 30 100000 45

I only want to normalize the Age, Income, and Height columns but my above method does not work becuase of the string data in the name state and gender columns.

cant you use try: .... except ValueError: ...... pattern. You can pass the normalization function on string columns. There might be more more 'pandastic' way of doing though. — user2707389
– user2707389, Commented Feb 18, 2015 at 5:51

desertnaut · Accepted Answer · 2022-03-15 15:18:02Z

You can perform operations on a sub set of rows or columns in pandas in a number of ways. One useful way is indexing:

# Assuming same lines from your example cols_to_norm = ['Age','Height'] survey_data[cols_to_norm] = survey_data[cols_to_norm].apply(lambda x: (x - x.min()) / (x.max() - x.min()))

This will apply it to only the columns you desire and assign the result back to those columns. Alternatively you could set them to new, normalized columns and keep the originals if you want.

People who read this answer be careful, the formula for normalization is wrong. The mean should be changed to min in the lambda function.
Yep. If what you are after is feature scaling each column [0,1] then, as @Amin says, use (x - x.min()) / (x.max() - x.min()) as the formula
is there really no built in function that just does this automatically?

Yaron · Accepted Answer · 2018-10-23 09:51:13Z

I think it's better to use 'sklearn.preprocessing' in this case which can give us much more scaling options. The way of doing that in your case when using StandardScaler would be:

from sklearn.preprocessing import StandardScaler cols_to_norm = ['Age','Height'] surveyData[cols_to_norm] = StandardScaler().fit_transform(surveyData[cols_to_norm])

Alvaro Silvino · Accepted Answer · 2016-08-31 22:20:40Z

Simple way and way more efficient:
Pre-calculate the mean:
dropna() avoid missing data.

mean_age = survey_data.Age.dropna().mean() max_age = survey_data.Age.dropna().max() min_age = survey_data.Age.dropna().min() dataframe['Age'] = dataframe['Age'].apply(lambda x: (x - mean_age ) / (max_age -min_age ))

this way will work...

seeker_after_truth · Accepted Answer · 2022-02-02 19:01:07Z

I think it's really nice to use built-in functions

# Assuming same lines from your example from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler() cols_to_norm = ['Age','Height'] survey_data[cols_to_norm] = scaler.fit_transform(survey_data[cols_to_norm])

Benjamin Ziepert · Accepted Answer · 2022-05-30 21:59:32Z

MinMax normalize all numeric columns with minmax_scale

import numpy as np from sklearn.preprocessing import minmax_scale # cols = ['Age', 'Height'] cols = df.select_dtypes(np.number).columns df[cols] = minmax_scale(df[cols])

Note: Keeps index, column names or non-numerical variables unchanged.

HK boy · Accepted Answer · 2019-11-27 17:22:39Z

import pandas as pd import numpy as np # let Dataset here be your data# from sklearn.preprocessing import MinMaxScaler minmax = MinMaxScaler() for x in dataset.columns[dataset.dtypes == 'int64']: Dataset[x] = minmax.fit_transform(np.array(Dataset[I]).reshape(-1,1))

Collectives™ on Stack Overflow

How can I normalize the data in a range of columns in my pandas dataframe

6 Answers 6

3 Comments

Comments

Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

3 Comments

Comments

Comments

Comments

Comments

Comments

Linked

Related