I am trying to figure out what data type my column in a spark data frame is and manipulate the column based on that dedeuction.
Here is what I have so far:
import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.appName('MyApp').getOrCreate() df = spark.read.csv('Path To csv File',inferSchema=True,header=True) for x in df.columns: if type(x) == 'integer': print(x+": inside if loop") The print(x+": inside if loop") statement never seems to get executed but I am sure there are several columns that are integer data type. What am I missing here?