We have a job running in CI that verifies the output of an ert batch run. The job also includes examples of failing tests which produce a backtrace. The job tests if the error message output matches our expectation.

This has worked well with Emacs 28.2 with a simple call to `diff`, checking if the error message is exactly the same. But after upgrading to Emacs 29.3 the error message changes between runs.

Emacs 28.2:

 Loading /solution/example-syntax-error.el (source)...
 Debugger entered--Lisp error: (void-variable DEF&*@)
 eval-buffer(#<buffer > nil "/solution/exampl..." nil t) ; Reading at buffer position 41
 load-with-code-conversion("/solution/exampl..." "/solution/exampl..." nil nil)
 load("/solution/exampl..." nil nil t)
 load-file("example-syntax-error.el")
 eval-buffer(#<buffer > nil "/solution/exampl..." nil t) ; Reading at buffer position 151
 load-with-code-conversion("/solution/exampl..." "/solution/exampl..." nil t)
 load("/solution/exampl..." nil t)
 command-line-1(("-l" "ert" "-l" "/solution/exampl..." "-f" "ert-run-tests-batch-and-exit"))
 command-line()
 normal-top-level()

Emacs 29.3:

 Loading /solution/example-syntax-error.el (source)...
 Error: void-variable (DEF&*@)
 mapbacktrace(#f(compiled-function (evald func args flags) #<bytecode -0x77b476f04510f1b>)) <--- the bytecode output changes
 debug-early-backtrace()
 debug-early(error (void-variable DEF&*@))
 eval-buffer(#<buffer > nil "/solution/example-syntax-error.el" nil t)
 load-with-code-conversion("/solution/example-syntax-error.el" "/solution/example-syntax-error.el" nil nil)
 load("/solution/example-syntax-error.el" nil nil t)
 load-file("example-syntax-error.el")
 eval-buffer(#<buffer > nil "/solution/example-syntax-error-test.el" nil t)
 load-with-code-conversion("/solution/example-syntax-error-test.el" "/solution/example-syntax-error-test.el" nil t)
 load("/solution/example-syntax-error-test.el" nil t)
 command-line-1(("-l" "ert" "-l" "/solution/example-syntax-error-test.el" "-f" "ert-run-tests-batch-and-exit"))
 command-line()
 normal-top-level()
 Symbol's value as variable is void: DEF&*@

I've traced back the problem to the introduction of `debug-early.el`, which is now always used in batch mode. [This lambda][1] changes the output every run of the `#<bytecode -0x77b476f04510f1b>` part, line 3 of the quote above.

So far I've only found the `backtrace-on-error-noninteractive` variable to deactivate the backtrace, which is not what we want because we don't set that flag in production, making the test less useful.

I've also tried to set the variable `debugger`, but that doesn't seem to have an effect. (I later saw that it's already set to `debug` in batch mode, so relevant code doesn't seem to care about that variable.)

Is there any way to make the output the same on every run again?


 [1]: https://github.com/emacs-mirror/emacs/blob/2c201bbba5c43328979bf139330684cacfa074f3/lisp/emacs-lisp/debug-early.el#L60