Given an RDD with several key-value pairs, where each value is actually a list of values, how do I split the value lists so that I end up with simple key-value pairs?
from pyspark import SparkConf, SparkContext conf = SparkConf() sc = SparkContext(conf=conf) foo = sc.parallelize([(0,[1,1,4]),(1,[3,5])]) bar = foo.map(magic) bar.collect() >>>>[(0,1),(0,1),(0,4),(1,3),(1,5)] What would magic look like to achieve what I want?
Figured it out:
bar=foo.flatMap(lambda l: [(l[0], value) for value in l[1]]) I realize that it is a rather simple problem and solution, but I'll leave it up in case anyone else is struggling starting out with pyspark.
Python lets you combine arbitrarily many generator expressions, essentially letting you "unwrap" a nested structure like this. Each "layer" will be a new for _ in _
lambda l: [(key, value) for (key, values) in l for value in values] >>> l = [(0,[1,1,4]),(1,[3,5])] >>> [(key, value) for (key, values) in l for value in values] [(0, 1), (0, 1), (0, 4), (1, 3), (1, 5)] 
