Return to Answer

Added More details about PyPy

edited Apr 16, 2019 at 14:34

Case 1.)Case 1.) If you have pure Python code (<= Python Language version 3.5, 3.6 has "beta level support") which only relies on pure Python modules, you can embrace SRP everywhere and use PyPy to run it. PyPy (https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html) is a Python interpreter which has a Just in Time Compiler (JIT) and can remove function call overhead as long as it has sufficient time to "warm up" by tracing the executed code (a few seconds IIRC). **

Case 2.)Case 2.) If you have Python code which uses C extensions (or relies on libraries which use C extensions), PyPy may or may not be useful depending on how they're written. See http://doc.pypy.org/en/latest/extending.html for details, but the summary is that CFFI has minimal overhead while CTypes is slower (using it with PyPy may be even slower than CPython)

I'm scared of being "locked into" Cython, whereas any code written for PyPy can run under CPython as well.

** Some extra notes on PyPy in Production

Be very careful about making any choices that have the practical effect of "locking you in" to PyPy in a large codebase. Because some (very popular and useful) third party libraries do not play nice for reasons mentioned earlier, it can cause very difficult decisions later if you realize you need one of those libraries. My experience is primarily in using PyPy to speed up some (but not all) microservices which are performance sensitive in an company environment where it adds negligible complexity to our production environment (we already have multiple languages deployed, some with different major versions like 2.7 vs 3.5 running anyways).

I have found using both PyPy and CPython regularly forced me to write code which only relies on guarantees made by the language specification itself, and not on implementation details which are subject to change at any time. You may find thinking about such details to be an extra burden, but I found it valuable in my professional development, and I think it is "healthy" for the Python ecosystem as a whole.

Case 1.) If you have pure Python code (<= Python Language version 3.5, 3.6 has "beta level support") which only relies on pure Python modules, you can embrace SRP everywhere and use PyPy to run it. PyPy (https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html) is a Python interpreter which has a Just in Time Compiler (JIT) and can remove function call overhead as long as it has sufficient time to "warm up" by tracing the executed code (a few seconds IIRC).

Case 2.) If you have Python code which uses C extensions (or relies on libraries which use C extensions), PyPy may or may not be useful depending on how they're written. See http://doc.pypy.org/en/latest/extending.html for details, but the summary is that CFFI has minimal overhead while CTypes is slower (using it with PyPy may be even slower than CPython)

I'm scared of being "locked into" Cython, whereas any code written for PyPy can run under CPython as well.

Case 1.) If you have pure Python code (<= Python Language version 3.5, 3.6 has "beta level support") which only relies on pure Python modules, you can embrace SRP everywhere and use PyPy to run it. PyPy (https://morepypy.blogspot.com/2019/03/pypy-v71-released-now-uses-utf-8.html) is a Python interpreter which has a Just in Time Compiler (JIT) and can remove function call overhead as long as it has sufficient time to "warm up" by tracing the executed code (a few seconds IIRC). **

Case 2.) If you have Python code which uses C extensions (or relies on libraries which use C extensions), PyPy may or may not be useful depending on how they're written. See http://doc.pypy.org/en/latest/extending.html for details, but the summary is that CFFI has minimal overhead while CTypes is slower (using it with PyPy may be even slower than CPython)

I'm scared of being "locked into" Cython, whereas any code written for PyPy can run under CPython as well.

** Some extra notes on PyPy in Production

Source Link

answered Apr 13, 2019 at 20:51

Steven Jackson

First, some clarifications: Python is a language. There are several different interpreters which can execute code written in the Python language. The reference implementation (CPython) is usually what is being referenced when someone talks about "Python" as if it is an implementation, but it is important to be precise when talking about performance characteristics, as they can differ wildly between implementations.

How and where do we embrace the SRP without compromising performance in Python, as its inherent implementation directly impacts it?

If you are restricted to using the CPython interpreter, you can extract the slow functions into extensions written in C, which will be pre-compiled and not suffer from any interpreter overhead. You can still use SRP everywhere, but your code will be split between Python and C. Whether this is better or worse for maintainability than selectively abandoning SRP but sticking to only Python code depends on your team, but if you have performance critical sections of your code, it will undoubtably be faster than even the most optimized pure Python code interpreted by CPython. Many of Python's fastest mathematical libraries use this method (numpy and scipy IIRC). Which is a nice segue into Case 2...

Cython (https://cython.org/) is another option which I don't have as much experience with. I mention it for the sake of completeness so my answer can "stand on its own", but don't claim any expertise. From my limited usage, it felt like I had to work harder to get the same speed improvements i could get "for free" with PyPy, and if I needed something better than PyPy, it was just as easy to write my own C extension (which has the benefit if I re-use the code elsewhere or extract part of it into a library, all my code can still run under any Python Interpreter and is not required to be run by Cython).

I'm scared of being "locked into" Cython, whereas any code written for PyPy can run under CPython as well.