Handle corrupted files in spark load()

Question

I have a spark job that runs daily to load data from S3.

These data are composed of thousands of gzip files. However, in some cases, there is one or two corrupted files in S3, and it causes the whole spark_reader.load() task to fail.

An error occurred while calling o112.load. incorrect header check

Is there a way to just log these corrupted files and not break the loading ?

Current code :

def read_data(spark: SparkSession) -> DataFrame: spark_reader = spark.read.format(json) return spark_reader.load("s3://my_bucket/some_folder/")

We need you to show some code to be able to help you

Basile Starynkevitch
– Basile Starynkevitch

2025-11-26 07:23:53 +00:00
Commented Nov 26 at 7:23 — Basile Starynkevitch
– Basile Starynkevitch, Commented Nov 26 at 7:23

boyangeor · Accepted Answer · 2025-11-26 14:32:01Z

0

You can try the ignoreCorruptFiles option, documentation.

answered Nov 26 at 14:32

boyangeor

1,2061 gold badge9 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Handle corrupted files in spark load()

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related