25

PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this mode gets better performance by disabling view tracking and version counter bumps."

If I am just evaluating my model at test time (i.e. not training), is there any situation where torch.no_grad is preferable to torch.inference_mode? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn't surface as a runtime error then I assume it is indeed preferable to use torch.inference_mode).

More details on why inference mode was developed are mentioned in the PyTorch Developer Podcast.

Improve this question
1
  • Current link explaining the differences between these: Locally disabling gradient computation. In short, no_grad disables gradients but allows you to use the resulting values in gradient computations later, while inference_mode doesn't, so the advice is to use it in things like data processing and model evaluation. Commented Sep 16, 2024 at 8:50

1 Answer 1

Reset to default
27

Yes, torch.inference_mode is indeed preferable to torch.no_grad in all situations where inference mode does not throw a runtime error. Check here.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

FYI the first link has now rotted.
link is now removed
a more insightful answer: stackoverflow.com/a/74197846/10805680
@simplyPTA Both my question and the link in my answer already mention view tracking, the version counter, and the PyTorch podcast episode - the answer you link does not mention anything additional that is responsible for the performance improvement

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.