I've used spark to compute the PCA on a large dataset, now I have a spark dataframe with the following structure:
Row('pcaFeatures'=DenseVector(elem1,emlem2..)) where elem1,..., elemN are double numbers. I would like to transform it in a numpy matrix. Right now I'm using the following code:
numpymatrix = datapca.toPandas().as_Matrix() but I get a numpy Series with elements of type Object instead of a numeric matrix. Is there a way to get the matrix I need?
Your request makes sense only if the resulting data can fit into your main memory (i.e. you can safely use collect()); on the other hand, if this is the case, admittedly you have absolutely no reason to use Spark at all.
Anyway, given this assumption, here is a general way to convert a single-column features Spark dataframe (Rows of DenseVector) to a NumPy array using toy data:
spark.version # u'2.2.0' from pyspark.ml.linalg import Vectors import numpy as np # toy data: df = spark.createDataFrame([(Vectors.dense([0,45,63,0,0,0,0]),), (Vectors.dense([0,0,0,85,0,69,0]),), (Vectors.dense([0,89,56,0,0,0,0]) ,), ], ['features']) dd = df.collect() dd # [Row(features=DenseVector([0.0, 45.0, 63.0, 0.0, 0.0, 0.0, 0.0])), # Row(features=DenseVector([0.0, 0.0, 0.0, 85.0, 0.0, 69.0, 0.0])), # Row(features=DenseVector([0.0, 89.0, 56.0, 0.0, 0.0, 0.0, 0.0]))] np.asarray([x[0] for x in dd]) # array([[ 0., 45., 63., 0., 0., 0., 0.], # [ 0., 0., 0., 85., 0., 69., 0.], # [ 0., 89., 56., 0., 0., 0., 0.]]) 