Skip to content

CI: add pre-commit action, include pyupgrade #36471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Sep 22, 2020
Merged

CI: add pre-commit action, include pyupgrade #36471

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8197c61
add CI pre-commit check, include pyupgrade, remove unnecessary 'langu…
MarcoGorelli Sep 19, 2020
1a6b477
upgrade doc/sphinxext/announce.py
MarcoGorelli Sep 19, 2020
e8135b4
remove doubled checks
MarcoGorelli Sep 19, 2020
f4788fa
update deprecated repo
MarcoGorelli Sep 19, 2020
27e285e
add flake8-rst back to code_checks
MarcoGorelli Sep 19, 2020
1e5bc80
don't remove from code checks yet
MarcoGorelli Sep 21, 2020
9c98406
:memo: update code standards
MarcoGorelli Sep 21, 2020
635b8d7
double-backtick
MarcoGorelli Sep 21, 2020
e2a334a
:heavy_plus_sign: add pyupgrade to environment.yml
MarcoGorelli Sep 21, 2020
b8aea46
:heavy_plus_sign: add pre-commit to environment.yml
MarcoGorelli Sep 21, 2020
5b6faa1
generate pip reqs from conda
MarcoGorelli Sep 22, 2020
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .github/workflows/pre-commit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: pre-commit

on:
pull_request:
push:
branches: [master]

jobs:
pre-commit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: pre-commit/action@v2.0.0
14 changes: 7 additions & 7 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,30 +3,30 @@ repos:
rev: 19.10b0
hooks:
- id: black
language_version: python3
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already specified here: https://github.com/psf/black/blob/master/.pre-commit-hooks.yaml
- repo: https://gitlab.com/pycqa/flake8
rev: 3.8.3
hooks:
- id: flake8
language: python_venv
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, they already specify this here https://gitlab.com/pycqa/flake8/-/blob/master/.pre-commit-hooks.yaml

and according to pre-commit docs https://pre-commit.com/

The python_venv language is now an alias to python
additional_dependencies: [flake8-comprehensions>=3.1.0]
- id: flake8
name: flake8-pyx
language: python_venv
files: \.(pyx|pxd)$
types:
- file
args: [--append-config=flake8/cython.cfg]
- id: flake8
name: flake8-pxd
language: python_venv
files: \.pxi\.in$
types:
- file
args: [--append-config=flake8/cython-template.cfg]
- repo: https://github.com/pre-commit/mirrors-isort
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is deprecated, see https://github.com/pre-commit/mirrors-isort

This mirror repository is deprecated, use isort directly.
rev: v5.2.2
- repo: https://github.com/PyCQA/isort
rev: 5.2.2
hooks:
- id: isort
language: python_venv
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already specified in https://github.com/PyCQA/isort/blob/develop/.pre-commit-hooks.yaml
exclude: ^pandas/__init__\.py$|^pandas/core/api\.py$
- repo: https://github.com/asottile/pyupgrade
rev: v2.7.2
hooks:
- id: pyupgrade
args: [--py37-plus]
38 changes: 0 additions & 38 deletions ci/code_checks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,31 +48,6 @@ fi
### LINTING ###
if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then

echo "black --version"
Copy link
Contributor

@jreback jreback Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how are these no longer needed?
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They're run during the pre-commit CI check, see https://github.com/pandas-dev/pandas/pull/36471/checks?check_run_id=1137265216 . Could always duplicate them just in case if you like
Copy link
Contributor

@jreback jreback Sep 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but do we run the pre-commit checks on CI? i think have a script that is separate from the pre-commit is needed as well (it can duplicate or be called by pre-commit), but key is to have everything in sync.
Copy link
Contributor

@dsaxton dsaxton Sep 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two cents: I think keeping this for a time just to ensure it never disagrees with pre-commit makes sense, then can remove later once we're fairly sure they're identical (also makes it easier to revert if misconfigured)
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but do we run the pre-commit checks on CI?

That's what this PR adds - if you go down to "All checks have passed" and expand, you'll see an extra GitHub Action called "pre-commit". If you expand that one further, it says

Run pre-commit/action@v2.0.0 install pre-commit Cache Size: ~21 MB (21895398 B) /bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/f09139a0-b8b5-427f-bc76-10b2403b8816/cache.tzst -P -C /home/runner/work/pandas/pandas /opt/hostedtoolcache/Python/3.8.5/x64/bin/pre-commit run --show-diff-on-failure --color=always --all-files black....................................................................Passed flake8...................................................................Passed flake8-pyx...............................................................Passed flake8-pxd...............................................................Passed isort....................................................................Passed pyupgrade................................................................Passed
Copy link
Member

@alimcmaster1 alimcmaster1 Sep 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two cents: I think keeping this for a time just to ensure it never disagrees with pre-commit makes sense, then can remove later once we're fairly sure they're identical (also makes it easier to revert if misconfigured)

This seems like a good approach! Also would be good to check that github actions CI fails if any of these pre commit stages doesn’t pass.
Copy link
Member Author

@MarcoGorelli MarcoGorelli Sep 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure - have added the ci/code checks back in for now then

Also would be good to check that github actions CI fails if any of these pre commit stages doesn’t pass.

From my experience using this in my own repo (https://github.com/nbQA-dev/nbQA) that's what happens
Copy link
Contributor

@dsaxton dsaxton Sep 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we open an issue to remember to deduplicate these checks later?
Copy link
Member

@simonjayhawkins simonjayhawkins Sep 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure
Copy link
Contributor

@dsaxton dsaxton Sep 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#36642
black --version

MSG='Checking black formatting' ; echo $MSG
black . --check
RET=$(($RET + $?)) ; echo $MSG "DONE"

# `setup.cfg` contains the list of error codes that are being ignored in flake8

echo "flake8 --version"
flake8 --version

# pandas/_libs/src is C code, so no need to search there.
MSG='Linting .py code' ; echo $MSG
flake8 --format="$FLAKE8_FORMAT" .
RET=$(($RET + $?)) ; echo $MSG "DONE"

MSG='Linting .pyx and .pxd code' ; echo $MSG
flake8 --format="$FLAKE8_FORMAT" pandas --append-config=flake8/cython.cfg
RET=$(($RET + $?)) ; echo $MSG "DONE"

MSG='Linting .pxi.in' ; echo $MSG
flake8 --format="$FLAKE8_FORMAT" pandas/_libs --append-config=flake8/cython-template.cfg
RET=$(($RET + $?)) ; echo $MSG "DONE"

echo "flake8-rst --version"
flake8-rst --version

Expand Down Expand Up @@ -132,19 +107,6 @@ if [[ -z "$CHECK" || "$CHECK" == "lint" ]]; then
fi
RET=$(($RET + $?)) ; echo $MSG "DONE"

echo "isort --version-number"
isort --version-number

# Imports - Check formatting using isort see setup.cfg for settings
MSG='Check import format using isort' ; echo $MSG
ISORT_CMD="isort --quiet --check-only pandas asv_bench scripts web"
if [[ "$GITHUB_ACTIONS" == "true" ]]; then
eval $ISORT_CMD | awk '{print "##[error]" $0}'; RET=$(($RET + ${PIPESTATUS[0]}))
else
eval $ISORT_CMD
fi
RET=$(($RET + $?)) ; echo $MSG "DONE"

fi

### PATTERNS ###
Expand Down
1 change: 0 additions & 1 deletion doc/sphinxext/announce.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- encoding:utf-8 -*-
"""
Script to generate contributor and pull request lists

Expand Down