2

I'm trying to cache the Python installation and dependencies in "Self-hosted" GitHub Action, the problem is the restore of cached pip is taking too long and results in timeout, and I could not figure out why.
What I checked:
-> Issue caching python dependencies in GitHub Actions
-> Github Actions don't reuse cache
-> Cache is not being correctly loaded in Github actions

in addition to the official Cache repo Docs on GitHub. below is the config used

- name: Set up Python uses: actions/setup-python@v4 id: python311 with: python-version: '3.11.8' cache: 'pip' cache-dependency-path: 'modules/**/requirements*.txt' - name: Cache Python dependencies uses: actions/[email protected] with: path: ~/.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }} restore-keys: | ${{ runner.os }}-pip- - name: Cache Python interpreter uses: actions/[email protected] with: path: ${{ env.pythonLocation }} #/opt/hostedtoolcache/Python/3.11.8 key: ${{ runner.os }}-python-${{ hashFiles('/opt/hostedtoolcache/Python/3.11.8') }} restore-keys: | ${{ runner.os }}-python- - name: Install requirements run: | mkdir -p .output pip install -r modules/localfiles/requirements.test.txt

The output:

Version 3.11.8 was not found in the local cache Version 3.11.8 is available for downloading Download from "https://github.com/actions/python-versions/releases/download/3.11.8-7809691605/python-3.11.8-linux-20.04-x64.tar.gz" Extract downloaded archive /usr/bin/tar xz --warning=no-unknown-keyword -C /runner/_work/_temp/ee3bccf9-3f77-4c32-97e6-429f1d6adb00 -f /runner/_work/_temp/9f59a8fb-fb9d-4d39-b3df-ffb7b7288fac Execute installation script Check if Python hostedtoolcache folder exist... Creating Python hostedtoolcache folder... Create Python 3.11.8 folder Copy Python binaries to hostedtoolcache folder Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action) Upgrading pip... Looking in links: /tmp/tmpr1ai5afl Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages (65.5.0) Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages (24.0) Collecting pip Downloading pip-24.0-py3-none-any.whl.metadata (3.6 kB) Downloading pip-24.0-py3-none-any.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 299.7 kB/s eta 0:00:00 Installing collected packages: pip Successfully installed pip-24.0 Create complete file Successfully set up CPython (3.11.8) /opt/hostedtoolcache/Python/3.11.8/x64/bin/pip cache dir /home/runner/.cache/pip Received 0 of 57628253 (0.0%), 0.0 MBs/sec Received 25165824 of 57628253 (43.7%), 12.0 MBs/sec Received 53433949 of 57628253 (92.7%), 17.0 MBs/sec Received 53433949 of 57628253 (92.7%), 12.7 MBs/sec Received 53433949 of 57628253 (92.7%), 10.2 MBs/sec Received 53433949 of 57628253 (92.7%), 8.5 MBs/sec Received 53433949 of 57628253 (92.7%), 7.3 MBs/sec Received 53433949 of 57628253 (92.7%), 6.4 MBs/sec Received 53433949 of 57628253 (92.7%), 5.7 MBs/sec Received 53433949 of 57628253 (92.7%), 5.1 MBs/sec Received 53433949 of 57628253 (92.7%), 4.6 MBs/sec Received 53433949 of 57628253 (92.7%), 4.2 MBs/sec Received 53433949 of 57628253 (92.7%), 3.9 MBs/sec Received 53433949 of 57628253 (92.7%), 3.6 MBs/sec Received 53433949 of 57628253 (92.7%), 3.4 MBs/sec Received 53433949 of 57628253 (92.7%), 3.2 MBs/sec Received 53433949 of 57628253 (92.7%), 3.0 MBs/sec Received 53433949 of 57628253 (92.7%), 2.8 MBs/sec Received 53433949 of 57628253 (92.7%), 2.7 MBs/sec Received 53433949 of 57628253 (92.7%), 2.5 MBs/sec Received 53433949 of 57628253 (92.7%), 2.4 MBs/sec Received 53433949 of 57628253 (92.7%), 2.3 MBs/sec Received 53433949 of 57628253 (92.7%), 2.2 MBs/sec Received 53433949 of 57628253 (92.7%), 2.1 MBs/sec Received 53433949 of 57628253 (92.7%), 2.0 MBs/sec Received 53433949 of 57628253 (92.7%), 2.0 MBs/sec Received 53433949 of 57628253 (92.7%), 1.9 MBs/sec Received 53433949 of 57628253 (92.7%), 1.8 MBs/sec Received 53433949 of 57628253 (92.7%), 1.8 MBs/sec Received 53433949 of 57628253 (92.7%), 1.7 MBs/sec Received 53433949 of 57628253 (92.7%), 1.6 MBs/sec Received 53433949 of 57628253 (92.7%), 1.6 MBs/sec Received 53433949 of 57628253 (92.7%), 1.5 MBs/sec Received 53433949 of 57628253 (92.7%), 1.5 MBs/sec Received 53433949 of 57628253 (92.7%), 1.5 MBs/sec Received 53433949 of 57628253 (92.7%), 1.4 MBs/sec Received 53433949 of 57628253 (92.7%), 1.4 MBs/sec Received 53433949 of 57628253 (92.7%), 1.3 MBs/sec Received 53433949 of 57628253 (92.7%), 1.3 MBs/sec Received 53433949 of 57628253 (92.7%), 1.3 MBs/sec Received 53433949 of 57628253 (92.7%), 1.2 MBs/sec Received 53433949 of 57628253 (92.7%), 1.2 MBs/sec Received 53433949 of 57628253 (92.7%), 1.2 MBs/sec Received 53433949 of 57628253 (92.7%), 1.2 MBs/sec Received 53433949 of 57628253 (92.7%), 1.1 MBs/sec Received 53433949 of 57628253 (92.7%), 1.1 MBs/sec Received 53433949 of 57628253 (92.7%), 1.1 MBs/sec Received 53433949 of 57628253 (92.7%), 1.1 MBs/sec Received 53433949 of 57628253 (92.7%), 1.0 MBs/sec Received 53433949 of 57628253 (92.7%), 1.0 MBs/sec Received 53433949 of 57628253 (92.7%), 1.0 MBs/sec Received 53433949 of 57628253 (92.7%), 1.0 MBs/sec Received 53433949 of 57628253 (92.7%), 1.0 MBs/sec Received 53433949 of 57628253 (92.7%), 0.9 MBs/sec Received 53433949 of 57628253 (92.7%), 0.9 MBs/sec Received 53433949 of 57628253 (92.7%), 0.9 MBs/sec Received 53433949 of 57628253 (92.7%), 0.9 MBs/sec Received 53433949 of 57628253 (92.7%), 0.9 MBs/sec Received 53433949 of 57628253 (92.7%), 0.9 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.8 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.7 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.6 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Received 53433949 of 57628253 (92.7%), 0.5 MBs/sec Warning: Failed to restore: The operation cannot be completed in timeout. pip cache is not found
Improve this question
2
  • 2
    You shouldn't have to do manual caching outside of setup-python, it should do the caching for you already. Commented Mar 25, 2024 at 12:52
  • @BenjaminW. Even if I remove the caching step it keeps freezing Commented Mar 27, 2024 at 15:48

1 Answer 1

Reset to default
0

I've opened an issue in GitHub, and got a reply like this:

The performance of actions/setup-python action on your self-hosted runners can be influenced by a few key factors: - Network bandwidth: Make sure that your network bandwidth is sufficient for the number of concurrent workflows. - Disk space: Ensure your self-hosted runner has enough disk space to store the downloaded Python versions. - Resource allocation: Check if other processes on your self-hosted runner machine are using significant system resources (CPU, memory, disk I/O). Additionally, the setup-python action caches the initially installed Python version on your runner for future use. In subsequent runs with the identical Python version, the action retrieves the cached version, bypassing the installation process and speeding up execution as no further downloads are necessary. Please review these aspects on your self-hosted runner. If the issue persists, kindly provide us with more detailed logs or a public repository to facilitate further investigation.

based on this answer I looked back at the Docs for Linux, and in doing so realized I needed to add RUNNER_TOOL_CACHE which reduced the time to nearly 0 seconds

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.