PySpark Dropping Columns Issues

Question

The goal of my code is to try to drop a column each time it shows up. I know there is a way to drop columns without using a for loop. The reason that method does not work is that the columns are dynamic. The problem is that the .drop command is not dropping the column indicated. So here is some pseudocode.

for column_name in column_name_list: # create data_frame1 with the column name # join data_frame with other data_frame2 # Here I drop column_name in data_frame1 data_frame = data_frame.drop(column_name)

The problem is after the drop, the column_name is re-appearing during the second iteration. My guess is that I am dropping the column on a copy and it's not "saving" the data_frame with the dropped column. Thank you for all the help.

hamza tuna · Accepted Answer · 2018-06-29 06:21:08Z

Maybe you have 2 columns with same name after join. If you join with same name You can easly join in this way easly:

dataframe1.join(dataframe2, col_name) # no need to dataframe1.col_name == dataframe2.col_name

if You already do and code above not working(i use this code and worked) you can use :

 data_frame.select(*set(dataframe.columns) - set(column_names_list))

Collectives™ on Stack Overflow

PySpark Dropping Columns Issues

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related