1

We have built an MVP in Python Data ecosystem, but the performance is much less than desired.

Our approach:

  • multiprocessing module for task parallelisation (which is awkwardly done in Python - pickle/unpickle each task)
  • scikit-learn library
  • pandas for data transport
  • it's not about dealing with big data or training models on large data, after profiling we figured out most of the time is taken by a regular computing tasks, but we are doing a lot.

Hence:

  1. We wonder what else can we use to make computing faster? After some research number of candidates popped up: Julia, Lua, Rust etc.
    • Requirements:
      • FASTER
      • Parallelisation support by design (not some workaround like Python does)
      • Existence of scientific libraries, not all of Python but major selection
      • Compiled (I guess)
  2. But before we move to another platform - we wonder if there's a way of speeding up existing Python code?
Improve this question

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.