How to show full column content in a Spark Dataframe?

Question

I am using spark-csv to load data into a DataFrame. I want to do a simple query and display the content:

val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("my.csv") df.registerTempTable("tasks") results = sqlContext.sql("select col from tasks"); results.show()

The col seems truncated:

scala> results.show(); +--------------------+ | col| +--------------------+ |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:15:...| |2015-11-06 07:15:...| |2015-11-16 07:15:...| |2015-11-16 07:21:...| |2015-11-16 07:21:...| |2015-11-16 07:21:...| +--------------------+

How do I show the full content of the column?

Dennis Jaheruddin · Accepted Answer · 2025-06-05 14:07:01Z

582

results.show(20, false) will not truncate, note that Python would require False rather than false which is valid in Scala/Java/Spark shell.

Check the source

20 is the default number of rows displayed when show() is called without any arguments.

edited Jun 5 at 14:07

Dennis Jaheruddin

21.6k8 gold badges73 silver badges132 bronze badges

answered Nov 16, 2015 at 19:24

TomTom101

6,9413 gold badges27 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

xv70 Over a year ago

Not OP but this is indeed the right answer : Minor correction, boolean should be False, not false.

drewrobb Over a year ago

It would be "False" in python, but "false" in scala/java

Luca Gibelli Over a year ago

it's false (not False) in spark-shell

JMess Over a year ago

the equivalent for writing to stream in console mode is dataFrame.writeStream.outputMode("append").format("console").option("truncate", "false").start()

Bikash Gyawali Over a year ago

what is so special about 20? Why 20?

|

Shubham Chaudhary · Accepted Answer · 2019-06-04 20:57:41Z

71

If you put results.show(false) , results will not be truncated

edited Jun 4, 2019 at 20:57

Shubham Chaudhary

51.6k9 gold badges84 silver badges81 bronze badges

answered Apr 8, 2016 at 19:02

Narendra Parmar

1,4091 gold badge14 silver badges18 bronze badges

5 Comments

Mogsdad Over a year ago

I imagine that the comment on TomTom101's answer about false applies here, too.

Jai Prakash Over a year ago

@Narendra Parmar the syntax should be results.show(20, False). The one you have mentioned will give error.

Narendra Parmar Over a year ago

@ Jai Prakash , i have given this answer for scala and you are talking about python,

Jai Prakash Over a year ago

@NarendraParmar sorry you are correct. In scala both the options are valid. results.show(false) and results.show(20, false)

Doug_Ivison Over a year ago

@JaiPrakash -- in ASA, "false" has to have a capital f: "False" is ok, but "false" gives an error.

MoeChen · Accepted Answer · 2017-02-05 01:21:24Z

43

Below code would help to view all rows without truncation in each column

df.show(df.count(), False)

answered Feb 5, 2017 at 1:21

MoeChen

7896 silver badges6 bronze badges

3 Comments

WestCoastProjects Over a year ago

same questio i asked the prior answerer: does this cause df to be collected twice?

MoeChen Over a year ago

@javadba yes, I think count() will go through df once, and show() will collect df twice.

Omkar Neogi Over a year ago

As an alternative, you could give a very large number as the first parameter instead of df.count() in order to save on the requirement to persist. For example, if the row count of df is 1000, you could do df.show(1000000, false) and it will work. Tried the following and it worked: scala> println(df.count) res2: Long = 987 scala> df.show(990)

codeaperature · Accepted Answer · 2017-02-15 06:25:17Z

The other solutions are good. If these are your goals:

No truncation of columns,
No loss of rows,
Fast and
Efficient

These two lines are useful ...

 df.persist df.show(df.count, false) // in Scala or 'False' in Python

By persisting, the 2 executor actions, count and show, are faster & more efficient when using persist or cache to maintain the interim underlying dataframe structure within the executors. See more about persist and cache.

Mario · Accepted Answer · 2025-11-02 22:43:53Z

In Pyspark we can use:

df.show(truncate=False) this will display the full content of the columns without truncation.
df.show(5,truncate=False) this will display the full content of the first five rows.

Sai · Accepted Answer · 2017-10-31 15:12:39Z

12

results.show(20, False) or results.show(20, false) depending on whether you are running it on Java/Scala/Python

edited Oct 31, 2017 at 15:12

Sai

7131 gold badge7 silver badges26 bronze badges

answered Mar 8, 2017 at 5:40

Deepak Babu P R

1211 silver badge2 bronze badges

Comments

farrellw · Accepted Answer · 2020-06-10 19:55:22Z

The following answer applies to a Spark Streaming application.

By setting the "truncate" option to false, you can tell the output sink to display the full column.

val query = out.writeStream .outputMode(OutputMode.Update()) .format("console") .option("truncate", false) .trigger(Trigger.ProcessingTime("5 seconds")) .start()

ngenne · Accepted Answer · 2022-04-05 12:13:54Z

In Spark Pythonic way, remember:

if you have to display data from a dataframe, use show(truncate=False) method.
else if you have to display data from a Stream dataframe view (Structured Streaming), use the writeStream.format("console").option("truncate", False).start() methods with option.

Hope it could helps someone.

Ignacio Alorre · Accepted Answer · 2018-09-10 09:12:34Z

4

Within Databricks you can visualize the dataframe in a tabular format. With the command:

display(results)

It will look like

answered Sep 10, 2018 at 9:12

Ignacio Alorre

7,6558 gold badges65 silver badges104 bronze badges

1 Comment

unkind58 Over a year ago

how with display() show only, for example, first 5 rows?

Vyacheslav · Accepted Answer · 2020-04-01 19:37:23Z

In c# Option("truncate", false) does not truncate data in the output.

StreamingQuery query = spark .Sql("SELECT * FROM Messages") .WriteStream() .OutputMode("append") .Format("console") .Option("truncate", false) .Start();

Mario · Accepted Answer · 2025-11-02 22:45:17Z

Try

df.show(20,False)

Notice that if you do not specify the number of rows you want to show, it will show 20 rows but will execute all your dataframe which will take more time !

epic_last_song · Accepted Answer · 2016-11-25 20:16:32Z

3

try this command :

df.show(df.count())

answered Nov 25, 2016 at 20:16

epic_last_song

1531 silver badge6 bronze badges

3 Comments

Thota Kranthi Kumar Over a year ago

Try this: df.show(some no) will work but df.show(df.count()) will not work df.count gives output type long which is not accepted by df.show() as it accept integer type.

Thota Kranthi Kumar Over a year ago

Example use df.show(2000). It will retrieve 2000 rows

WestCoastProjects Over a year ago

does this cause df to be collected twice?

OneCricketeer · Accepted Answer · 2018-02-13 01:54:42Z

results.show(false) will show you the full column content.

Show method by default limit to 20, and adding a number before false will show more rows.

zero323 · Accepted Answer · 2018-11-20 16:55:45Z

3

results.show(20,false) did the trick for me in Scala.

edited Nov 20, 2018 at 16:55

zero323

331k108 gold badges982 silver badges958 bronze badges

answered Apr 16, 2018 at 18:32

SKA

673 bronze badges

Comments

onemanarmy · Accepted Answer · 2020-09-18 12:29:37Z

3

Tried this in pyspark

df.show(truncate=0)

answered Sep 18, 2020 at 12:29

onemanarmy

9310 bronze badges

Comments

Sarath Subramanian · Accepted Answer · 2021-01-13 04:41:39Z

PYSPARK

In the below code, df is the name of dataframe. 1st parameter is to show all rows in the dataframe dynamically rather than hardcoding a numeric value. The 2nd parameter will take care of displaying full column contents since the value is set as False.

df.show(df.count(),False)

SCALA

In the below code, df is the name of dataframe. 1st parameter is to show all rows in the dataframe dynamically rather than hardcoding a numeric value. The 2nd parameter will take care of displaying full column contents since the value is set as false.

df.show(df.count().toInt,false)

Mario · Accepted Answer · 2025-11-02 22:46:14Z

PYSPARK

ds.show(df.count(),truncate=0)

The first parameter helps us to show all records The second parameter will help for column expansion.

Note: observed a behaviour difference between using truncate=False and truncate=0, 0 actually expands the column data while False doesn't

Pritesh Kumar · Accepted Answer · 2019-12-10 01:53:37Z

Try this in scala:

df.show(df.count.toInt, false)

The show method accepts an integer and a Boolean value but df.count returns Long...so type casting is required

Collectives™ on Stack Overflow

How to show full column content in a Spark Dataframe?

18 Answers 18

9 Comments

5 Comments

3 Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

3 Comments

Comments

Comments

Comments

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

18 Answers 18

9 Comments

5 Comments

3 Comments

Comments

Comments

Comments

Comments

Comments

1 Comment

Comments

Comments

3 Comments

Comments

Comments

Comments

Comments

Comments

Comments

Linked

Related