WIP/CI/TST: Clean up of the tests script

[datapythonista]

Follow up of the conversation in #26932, and fixing some of the things that #24106 tried.

The main change is call pytest once instead of once for single and once for multi. This shouldn't be a problem since I add the parameter --dist=loadfile which should prevent tests in the same file running in parallel. I don't think running serial tests in parallel with tests in other files should be a problem.

Other things cleaned in the PR:

• Setting always the proxy (shouldn't hurt for not network tests, and makes things simpler)
• Using as many processes as cores, not 2
• Removed hack for when no tests were found (since we don't call only for single tests, there will always be tests to run)

So far testing, let's see how does this work in the CI.

[TomAugspurger]

Using dist=loadfile seems fine, assuming things are reasonable evenly spread across workers. I imagine that some of our files take much longer, so hopefully pytest-xdist is smart about scheduling.

[datapythonista]

I guess we may need to split some big test files, but I think Jeff was requesting that for some already, and it's probably good anyway. I remember the tests taking longer the last time I tried this.

[datapythonista]

Looks like couple of builds are indeed significantly longer here than when not using --dist=loadfile. I think should be easy to get the list of test files, with the amount time that each takes, and see if it makes sense to split something.

Then, I see that some tests are failing because the pandas module has more attributes than expected, 'hard_dependencies', 'dependency', 'missing_dependencies'. Not sure how this could be related to the changes here. Did anyone faced this before?

[TomAugspurger]

Other PRs are hitting that. I think there’s a test order issue. We should skip that test for now.

…

On Jun 19, 2019, at 17:14, Marc Garcia ***@***.***> wrote: Looks like couple of builds are indeed significantly longer here than when not using --dist=loadfile. I think should be easy to get the list of test files, with the amount time that each takes, and see if it makes sense to split something. Then, I see that some tests are failing because the pandas module has more attributes than expected, 'hard_dependencies', 'dependency', 'missing_dependencies'. Not sure how this could be related to the changes here. Did anyone faced this before? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jbrockmendel]

needs rebase

[jbrockmendel]

@datapythonista needs rebase

[TomAugspurger]

CI is hopefully fixed if you can merge master.

[datapythonista]

Thanks, but even if the CI is green we can't merge this, since some builds take much longer with the change. Need to find the time to figure out how to solve it. Will leave it red for now so this is not merged by mistake.

[jreback]

getting pretty stale, closing.