How do I create a new column has the count of all the row values that are greater than 0 in pyspark?

Don't know if there is a simpler SQL based solution or not, but it's pretty straight forward with a udf.

count_udf = udf(lambda arr: sum([1 for a in arr if a > 0]), IntegerType()) df.withColumn('check', count_udf(array('col1', 'col2', 'col3'))).show() 

Not sure if it'll handle nulls. Add null check (if a and a > 0) in udf if needed.

Idea: https://stackoverflow.com/a/42540401/496289

Your code shows you doing a sum of non-zero columns, not count. If you need sum then

count_udf = udf(lambda arr: sum([a for a in arr if a > 0]), IntegerType()) 

Create a new column array and filter the newly created column finally count the elements in the column.

Example:

df.show(10,False) #+----+----+----+ #|col1|col2|col3| #+----+----+----+ #|1 |2 |-3 | #|2 |null|5 | #+----+----+----+ df.withColumn("check",expr("size(filter(array(col1,col2), x -> x > 0))")).show(10,False) #+----+----+----+-----+ #|col1|col2|col3|check| #+----+----+----+-----+ #|1 |2 |-3 |2 | #|2 |null|5 |1 | #+----+----+----+-----+ 

You can use functools.reduce to sum the list of columns from df.columns if > 0 like this:

from pyspark.sql import functions as F from operator import add from functools import reduce df = spark.createDataFrame([ (1, 2, -3), (2, None, 5), (4, 4, 8), (1, 0, 9) ], ["col1", "col2", "col3"]) df = df.withColumn( "check", reduce(add, [F.when(F.col(c) > 0, 1).otherwise(0) for c in df.columns]) ) df.show() #+----+----+----+-----+ #|col1|col2|col3|check| #+----+----+----+-----+ #| 1| 2| -3| 2| #| 2|null| 5| 2| #| 4| 4| 8| 3| #| 1| 0| 9| 2| #+----+----+----+-----+