Linked Questions

I'm searching the way to add a column 'id' to my dataframe (dfProc) with sequencial numbers from 1 (or zero) to number of rows (in this example it has 10 rows but my df has variable rows). The ...

I am using pyspark 1.3.1, I need to generate unique id/number for each row in a dataframe. Since window functions are not available with Pyspark Version:1.3.1, I am not able to make use of rownumber ...

I have a Spark DataFrame (using PySpark 1.5.1) and would like to add a new column. I've tried the following without any success: type(randomed_hours) # => list # Create in Python and transform ...

I have a csv file; which i convert to DataFrame(df) in pyspark; after some transformation; I want to add a column in df; which should be simple row id (starting from 0 or 1 to N). I converted df in ...

From a PySpark SQL dataframe like name age city abc 20 A def 30 B How to get the last row.(Like by df.limit(1) I can get first row of dataframe into new dataframe). And how can I access the ...

In python or R, there are ways to slice DataFrame using index. For example, in pandas: df.iloc[5:10,:] Is there a similar way in pyspark to slice data based on location of rows?

Assuming I am having the following dataframe: dummy_data = [('a',1),('b',25),('c',3),('d',8),('e',1)] df = sc.parallelize(dummy_data).toDF(['letter','number']) And i want to create the following ...

Is there a way to append a dataframe horizontally to another one - assuming both have identical number of rows? This would be the equivalent of pandas concat by axis=1; result = pd.concat([df1, df4],...

random number generator SparkSQL ? For example: Netezza: sequence number mysql: sequence number Thanks.

I have a Dataframe with single column like shown below. Type 'BAT' 'BAT' 'BALL' 'BAT' 'BALL' 'BALL' To the above dataframe I have added a new column called 'const'. df = df.withColumn('const',F.lit(...

I have a python list (p_list) with 0 and 1 with as many elements as a spark dataframe that has one column only (all elements, are like: 'imaj7felb438l6hk', ....). And I am trying to add this list as ...