I haven't done this in the past or validate it but at first glance, although you're right that comparing absolute time elapsed across runs would be tricky, maybe it's worth focusing on the number of instructions/function calls during a run? I think that would only vary between processor/versions of dependencies, not depending on whether or not your VM has a noisy neighbor or some other process slows down a test run a smidge. If you have one less function call in your hot loop, you'll make it faster wherever/however it runs.

cProfile can dump binary reports, which you can probably store as artifacts from each build, and then you can read them/compare them.

I haven't done this in the past or validated it but at first glance, although you're right that comparing absolute time elapsed across runs would be tricky, maybe it's worth focusing on the number of instructions/function calls during a run? I think that would only vary between processor/versions of dependencies, not depending on whether or not your VM has a noisy neighbor or some other process slows down a test run a smidge. If you have one less function call in your hot loop, you'll make it faster wherever/however it runs.

cProfile can dump binary reports, which you can probably store as artifacts from each build, and then you can read them/compare them.