Skip to main content

Return to Answer

added 169 characters in body
Source Link
viper
viper
  • 2.2k
  • 5
  • 28
  • 33

I tried to use threading with python, there were issues with the Global Interpreter Lock. Ported code to use the multiprocessing module and it worked with 2x speedup, internally hadoop assigns as many mappers as there are cores in the cluster, hence multiprocessing is not the way to go if you need speed up. Multithreading if performed right might give some speedup

I tried to use threading with python, there were issues with the Global Interpreter Lock. Ported code to use the multiprocessing module and it worked with 2x speedup.

I tried to use threading with python, there were issues with the Global Interpreter Lock. Ported code to use the multiprocessing module, internally hadoop assigns as many mappers as there are cores in the cluster, hence multiprocessing is not the way to go if you need speed up. Multithreading if performed right might give some speedup

Source Link
viper
viper
  • 2.2k
  • 5
  • 28
  • 33

I tried to use threading with python, there were issues with the Global Interpreter Lock. Ported code to use the multiprocessing module and it worked with 2x speedup.