Spark add new column with value form previous some columns

Question

I have DataFrame like this:

+----------+---+ | code |idn| +----------+---+ | [I0478]| 0| | [B0527]| 1| | [C0798]| 2| | [C0059]| 3| | [I0767]| 4| | [I1001]| 5| | [C0446]| 6| +----------+---+

And i want to add new column to DataFrame

+----------+---+------+ | code |idn| item | +----------+---+------+ | [I0478]| 0| I0478| | [B0527]| 1| B0527| | [C0798]| 2| C0798| | [C0059]| 3| C0059| | [I0767]| 4| I0767| | [I1001]| 5| I1001| | [C0446]| 6| C0446| +----------+---+------+

Please help me do this!

user8707959 · Accepted Answer · 2017-10-02 10:51:17Z

3

Use []:

df.withColumn("item", df["item"][0])

answered Oct 2, 2017 at 10:51

user8707959

391 bronze badge

Sign up to request clarification or add additional context in comments.

2 Comments

Phong Nguyen Over a year ago

AnalysisException: u"Field name should be String Literal, but it's 0;"

Smile Over a year ago

@PhongNguyen I had the same issue even with []. Turned out my column was actually Struct rather than Array..

Dileep Unnikrishnan · Accepted Answer · 2020-03-03 04:18:33Z

So the problem will be evident if you look at the schema - the column you are trying to subset is not an array. So the solution is to .* expand the column.

df.select('code.*', 'idn')

Willie Cheng · Accepted Answer · 2020-03-03 05:03:01Z

python

import pandas as pd array = {'code': [['I0478'],['B0527'], ['C0798'], ['C0059'], ['I0767'], ['I1001'], ['C0446']], 'idn':[0, 1, 2, 3, 4, 5, 6]} df = pd.DataFrame(array) df['item'] = df.apply(lambda row: str(row.code).lstrip('[').rstrip(']').strip("'").strip(), axis= 1) print(df)

CeciCCCCC · Accepted Answer · 2018-11-12 16:53:13Z

df.withColumn("item", df["code"][0])

If the "item" column is Array type, if it's Struct of string, you may need to inspect the key of the element of item by df.select("code").collect()[0], see what key(string) it has.

Collectives™ on Stack Overflow

Spark add new column with value form previous some columns

4 Answers 4

2 Comments

Comments

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

1 Comment

Comments

Related