Suppose I have got a list of key-value pairs:
kvs = [('x', 0), ('a', 1)] Now I'd like to create a Spark Row from kvs with the same order of keys as in kvs.
How to do it in Python ?
- convert I to dict and use Row(**kvs)Harry– Harry2017-10-01 11:02:27 +00:00Commented Oct 1, 2017 at 11:02
- It does not preserve the order of the pairs.Michael– Michael2017-10-01 11:10:02 +00:00Commented Oct 1, 2017 at 11:10
- you can use OrderedDict stackoverflow.com/questions/38253385/…Harry– Harry2017-10-01 11:10:54 +00:00Commented Oct 1, 2017 at 11:10
Add a comment |
2 Answers
I haven't run it yet but may you check once I will edit after running if fails.
from pyspark.sql import Row kvs = [('x', 0), ('a', 1)] h = {} [h.update({k:v}) for k,v in kvs] row = Row(**h) 2 Comments
Michael
Thanks but it does not preserve the order of the pairs in
kvs,Harry
Check how to preserve order using OrderedDict stackoverflow.com/questions/38253385/…
You can:
from pyspark.sql import Row Row(*[k for k, _ in kvs])(*[v for _, v in kvs]) but in my opinion it is better to avoid Row whatsoever. Other than being a convenient class to represent local values fetched from the JVM backend, it has no special meaning in Spark. In almost every context:
tuple(v for _, v in kvs) is perfectly valid replacement for Row.
