Tensorflow inference very slow on first time run in docker

Question

I built a docker image for a server that does inference with tensorflow. I installed tensorflow-gpu with pip in the docker image. It works fine for my machine with titan x gpus. But when I ran the docker container on another machine with 1080 ti gpus. The first run becomes incredibly slow, takes about 90 seconds, usually it takes 7 seconds on the first run and 1 second in the following runs. I tried to set tf_cudnn_use_autotune to 0, and also mount a folder to save the cuda cache. But it doesn't really solve the problem. Any one has any suggestion?

rrcal · Accepted Answer · 2019-07-29 03:58:13Z

2

Here's a link. I find this.

After running TensorFlow once, the compiled kernels are cached by CUDA. If using a docker container, the data is not cached and the penalty is paid each time TensorFlow starts.

edited Jul 29, 2019 at 3:58

rrcal

3,7906 gold badges27 silver badges35 bronze badges

answered Jul 29, 2019 at 1:27

joyceye

213 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Andrey Over a year ago

Any idea, where these cached kernels can be located?

Collectives™ on Stack Overflow

Tensorflow inference very slow on first time run in docker

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related