Is there a way to get the column data type in pyspark?

Question

Has been discussed that the way to find the column datatype in pyspark is using df.dtypes get datatype of column using pyspark. The problem with this is that for datatypes like an array or struct you get something like array<string> or array<integer>.

Question: Is there a native way to get the pyspark data type? Like ArrayType(StringType,true)

user10168341 · Accepted Answer · 2018-08-01 23:55:27Z

36

Just use schema:

df.schema[column_name].dataType

answered Aug 1, 2018 at 23:55

user10168341

3763 silver badges2 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

snark Over a year ago

dict(df.dtypes)[column_name] also works (thanks @RobinL). But note you'll get the datatype as a string name rather than the formal type name; e.g., timestamp vs TimestampType.

Tomasz Gandor Over a year ago

@snark - you mean a str instead of a type object, like TimestampType()? Thanks for the comment, in the end I might want the string ;)

snark Over a year ago

Yes, as a string. See spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/…

Collectives™ on Stack Overflow

Is there a way to get the column data type in pyspark?

1 Answer 1

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Linked

Related