restart python 3.14 migration due bytecode invalidation in rc3

[h-vetinari]

I dislike bumping migration_number:s, because it makes the mechanics of "when/how does this migrator apply" much less obvious for maintainers.

However, in this case I think our hands are tied. Python packages have compiled bytecode (.pyc files) which contain a magic number that determines whether they're consumable for a given runtime. In 3.14.0rc3 this magic number was bumped again (after we had already waited for rc2 because that also bumped it).

This doesn't break the existing artefacts, but it invalidates their bytecode, meaning any user of py314 that downloads one of the packages with the old magic number will have to wait while a bunch of code gets recompiled, which will be quite visible as a regression of start-up time in a new environment.

Since we're expressly publishing builds intended to be compatible with 3.14 GA already, the only way to ensure that we have a clean set of packages is to rebuild everything. This is painful now, but will only get more painful in the future, so let's bite the bullet sooner rather than later.

Draft until conda-forge/python-feedstock#814 is in.

CC @conda-forge/core

[conda-forge-admin]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

[beckermr]

I wonder if we should wait a few weeks and simply migrate again when 314 is released?

Reissuing all of these PRs is a lot of work for everyone and clearly the release candidates are not stable enough for us to use in this way.

I don't think we can migrate early in the future if we are going to encounter these large rebuilds.

[h-vetinari]

I don't think we can migrate early in the future if we are going to encounter these large rebuilds.

I don't think it's so black and white. The current process worked fine for 3.12 and 3.13. The bump of the magic number this late is annoying, but since it's due to a crash, ultimately understandable. But I can bring this concern to the python discourse, that these bytecode invalidations have a big cost for us, and should be avoided in the rc phase.

[h-vetinari]

I wonder if we should wait a few weeks and simply migrate again when 314 is released?

FWIW, I'm against this. I think we should restart the current migration ASAP and then keep going as usual. There are no further rc's forthcoming.

[beckermr]

I don't think the fact that there are no more RCs means for sure we wouldn't have to rebuild again for the first release due to some other bug.

However, go ahead if you feel strongly.

[h-vetinari]

I don't think the fact that there are no more RCs means for sure we wouldn't have to rebuild again for the first release due to some other bug.

That is always true on a theoretical level, but in practice, the CPython people are aware of the cost, which is why they promise no ABI breaks (for example), and why we believe them. It's in everyone's interest for this social agreement to be respected (where no other concerns exist that have even higher cost).

But I'll definitely bring up the argument with the CPython folks that increasing the magic bytecode number almost has the same effect on conda-forge as an ABI break (i.e. requiring a complete rebuild, resp. restart of any ongoing migration).

[hmaarrfk]

I don’t really see this regression as bad enough to restart the migration.

Noarch python also suffer from this regression.

It sucks, but bytecode compilation right?

[h-vetinari]

As soon as conda-forge/python-feedstock#814 is merged (or if we add artefact persistence to that PR), we could do some timing tests of python 3.14.0 rc2 vs rc3, in an environment full of builds against the old magic number.

It's always helpful to have some concrete metrics for decision-making. Perhaps I'm way off, but my intuition is that for larger environments (of packages that are all being actually imported in the code under test), we're talking about potentially several minutes or longer being added to the first start up.

If we're in that ballpark, I don't think that'd be an acceptable regression TBH, made worse by the fact that a lot of rarely-rebuilt feedstocks will keep that regression for a long time.

PS. I commented in the discourse thread I linked in the OP, discussion is ongoing (basically: this shouldn't happen so late, but not doing it was considered worse).

[xhochy]

As soon as conda-forge/python-feedstock#814 is merged (or if we add artefact persistence to that PR), we could do some timing tests of python 3.14.0 rc2 vs rc3, in an environment full of builds against the old magic number.

I'm slightly positive towards merging this PR before the final release, but would like to see the numbers first as this PR comes at great cost and thus should really be based on strong numbers.

Also: noarch shouldn't be affected as the bytecode is not generated at build time?

[h-vetinari]

I did

mamba create -n timing -c conda-forge -c conda-forge/label/python_rc python=3.14.0rc2 pillow scipy matplotlib numpy 

created a little script to exercise the loading of bytecode by importing stuff

imports.py

# optimistic star imports from numpy import * from scipy import * from matplotlib import * from PIL import * # try to force exhaustive imports import numpy.dtypes import numpy.exceptions import numpy.f2py import numpy.fft import numpy.lib import numpy.lib.format import numpy.lib.mixins import numpy.lib.recfunctions import numpy.lib.scimath import numpy.lib.stride_tricks import numpy.lib.npyio import numpy.lib.introspect import numpy.lib.array_utils import numpy.linalg import numpy.ma import numpy.ma.extras import numpy.ma.mrecords import numpy.polynomial import numpy.polynomial.chebyshev import numpy.polynomial.hermite import numpy.polynomial.hermite_e import numpy.polynomial.laguerre import numpy.polynomial.legendre import numpy.polynomial.polynomial import numpy.random import scipy.cluster import scipy.cluster.hierarchy import scipy.cluster.vq import scipy.constants import scipy.datasets import scipy.fft import scipy.fftpack import scipy.integrate import scipy.interpolate import scipy.io import scipy.io.arff import scipy.io.matlab import scipy.io.wavfile import scipy.linalg import scipy.linalg.blas import scipy.linalg.cython_blas import scipy.linalg.cython_lapack import scipy.linalg.interpolative import scipy.linalg.lapack import scipy.misc import scipy.ndimage import scipy.odr import scipy.optimize import scipy.signal import scipy.signal.windows import scipy.sparse import scipy.sparse.csgraph import scipy.sparse.linalg import scipy.spatial import scipy.spatial.distance import scipy.spatial.transform import scipy.special import scipy.stats import scipy.stats.contingency import scipy.stats.distributions import scipy.stats.mstats import scipy.stats.qmc import scipy.stats.sampling import PIL.Image import PIL.ImageCms import matplotlib.backends._backend_agg import matplotlib.ft2font import matplotlib.pyplot import mpl_toolkits import pylab

measured the timing with rc2, then updated the same environment to rc3, and measured again. On linux I got

$ time python imports.py # before (rc2; byte code should not need recompilation) real 0m2.185s user 0m2.971s sys 0m0.259s # after (rc3; byte code gets recompiled) real 0m4.389s user 0m5.306s sys 0m0.230s 

and on windows

Measure-Command { python imports.py } [...] TotalSeconds : 1.7986322 # before (rc2; byte code should not need recompilation) TotalSeconds : 9.2766205 # after (rc3; byte code gets recompiled) 

So it's certainly measurable, but less than I had feared. On the other hand, I only had a pretty small-ish environment (many big packages aren't available for 3.14 yet), so in an environment that imports 10s or 100s of packages, this will probably be much more noticeable.

[hmaarrfk]

I think you can test on a "production" environment of yours:

(cd ${CONDA_PREFIX} && find . -type d -name "__pycache__" -exec rm -rf {} +) # Invalidate a bunch of system caches and stuffs echo 1 | sudo tee /proc/sys/vm/drop_caches >/dev/null sudo swapoff -a sudo swapon -a # Replace torch with your heavy package echo "without __pycache__" time python -c "import torch; import xarray; import pandas" # Reinvalidate system caches, but not the __pycache__ echo 1 | sudo tee /proc/sys/vm/drop_caches > /dev/null sudo swapoff -a sudo swapon -a echo "with __pycache__" time python -c "import torch; import xarray; import pandas" 

For me (linux 64 bit, modern processor, SATA SSD)

without __pycache__ real	0m3.236s user	0m4.305s sys	0m0.328s with __pycache__ real	0m1.830s user	0m2.903s sys	0m0.315s 

Its not so much that I don't like speedup, just given it is "limited" in use, AND that packages naturally get rebuilt, it should be of "limited" value given the added churn.

[h-vetinari]

Closing as not worth the hassle, given how comparatively minor the performance number indicate this to be.