pytest version: 6.0.1
The fix in #7571 (backported to 6.0.1) has a bug where it does a "set" instead of "setdefault" to the _initial_handler_level. So if there are multiple calls to caplog.set_level, the level will be restored to that of the one-before-last call, instead of the value before the test.
Will submit a fix for this shortly.