Skip to main content
edited tags
Link
pault
  • 43.7k
  • 17
  • 121
  • 161
Source Link
ozzboy
  • 2.7k
  • 9
  • 44
  • 72

Spark Data Frames - Check if column is of type integer

I am trying to figure out what data type my column in a spark data frame is and manipulate the column based on that dedeuction.

Here is what I have so far:

import pyspark from pyspark.sql import SparkSession spark = SparkSession.builder.appName('MyApp').getOrCreate() df = spark.read.csv('Path To csv File',inferSchema=True,header=True) for x in df.columns: if type(x) == 'integer': print(x+": inside if loop") 

The print(x+": inside if loop") statement never seems to get executed but I am sure there are several columns that are integer data type. What am I missing here?