It's happening because you are not applying partitionBy on key-value pair rdd. your rdd must need to be in key-value pair. Also you key type should be int. I don't have sample data for your hive table. so lets demonstrate the fact using below hive table:

I have created a below dataframe using hive table

now I wish to partition my dataframe and want to keep the similar keys in one partition. so I have converted my dataframe to rdd as you can only apply partitionBy on rdd for re-partitioning.

hence, we need to convert our rdd into key-value pair to use paritionBy

now you can see that rdd has been converted to key value pair and you can therefore distribute your data in partitions according to keys available.

using a paritionBY on key-value rdd now:

Lets take a look at the partitions. data is grouped and similar keys are stored into similar partitions now. two of them are empty.

Now lets say I want to custom partitioning my data. so I have created below function to keep keys '1' and '3' in similar partition.

as you can see now that keys 1 and 3 are stored in one partition and rest on other.

I hope this help. you can try to paritionBy your dataframe. make sure to convert it into key value pair and keeping key as type int.

It's happening because you are not applying partitionBy on key-value pair rdd. Your rdd must be in key-value pair. Also, your key type should be integer. I don't have sample data for your hive table. So let's demonstrate the fact using below hive table:

I have created a below dataframe using hive table :

Now, I wish to partition my dataframe and want to keep the similar keys in one partition. So, I have converted my dataframe to rdd as you can only apply partitionBy on rdd for re-partitioning.

Hence, we need to convert our rdd into key-value pair to use paritionBy

Now,you can see that rdd has been converted to key value pair and you can therefore distribute your data in partitions according to keys available.

Using a paritionBy on key-value rdd now:

Lets take a look at the partitions. Data is grouped and similar keys are stored into similar partitions now. Two of them are empty.

Now lets say I want to custom partitioning my data. So I have created below function to keep keys '1' and '3' in similar partition.

As you can see now that keys 1 and 3 are stored in one partition and rest on other.

I hope this helps. You can try to partitionBy your dataframe. Make sure to convert it into key value pair and keeping key as type integer.

I hope this help. you can try to paritionBy your dataframe. make sure to convert it into key value pair and keeping key as type int.

Now lets say I want to custom partitioning my data. so I have created below function to keep keys '1' and '3' in similar partition.

def partitionFunc(key): import random if key == 1 or key == 3: return 0 else: return random.randint(1,2) newrdd = keypair_rdd.partitionBy(5,lambda k: partitionFunc(int(k[0]))) >>> print("Partitions structure: {}".format(newrdd.glom().map(len).collect())) Partitions structure: [3, 3, 0, 0, 0] 

as you can see now that keys 1 and 3 are stored in one partition and rest on other.

I hope this help. you can try to paritionBy your dataframe. make sure to convert it into key value pair and keeping key as type int.