issue_comments

Ok, let's get this in. matplotlib-base and nodefaults should be quite uncontroversial. If mamba makes problems it's quickly removed.

I plan to add a whats new entry concerning the CI speed-up (#4672, #4685 & #4694) in #4694

See #4694

I thought that pytest-xdist was not an option as the plot tests are not thread safe (as mpl isn't). But looking again I think that pytest-xdist actually uses uses multiprocessing and not multithreading, so this might actually be worth a try.

Not sure how much it would help on CI, but it would be nice to get pytest-xdist working again for local testing (#3263)

using mamba and matplotlib-base should speed up the installation step by 2 to 5 minutes. If we are fine switching to the faster but maybe not-as-mature mamba this can be merged on green.

Seems like a great improvement to me!

I wasn't really able to get to the bottom of this. Still, using mamba and matplotlib-base should speed up the installation step by 2 to 5 minutes. If we are fine switching to the faster but maybe not-as-mature mamba this can be merged on green.

Why do we see that much of a speed-up once we downgrade numba on azure pipelines?

Sometimes it also works fine with numba 0.52... So unfortunately I don't know. My suspicion is that we get different CPUs by chance. I added a new step to our CI: cat /proc/cpuinfo (worked with gitbash on windows). Maybe this reveals something.

On my dualboot machine the test suite takes 23 min on windows and 15 min on linux. Thus, already quite a difference but not as large as on azure where the windows test seem to take about twice as long as the linux tests.

locally on windows I find no large difference between numba 0.51 and 0.52

that's really strange. Why do we see that much of a speed-up once we downgrade numba on azure pipelines?

there are about 750 xfailed tests in test_units.py

yes, I will have to update those. I think until the index refactor we can safely skip all tests that rely on units in indexes, which should improve the situation, and there might also be a few tests that were fixed by pint.

Edit: see #4685

What is the difference between pytest.mark.xfail and pytest.xfail?

I think pytest.mark.xfail is the official way to decorate test functions while pytest.xfail can be used in the function body to programmatically mark the test as expected failure (which allows more control than the mark)

locally on windows I find no large difference between numba 0.51 and 0.52, so that does not seem to be the root cause...

@keewis there are about 750 xfailed tests in test_units.py. xfail is the correct category but they take much longer than skip. Locally the tests take about 6 min 30 s using xfail but only 45 s using skip. On azure the difference is probably even bigger. Would it be an option to use skip instead? Of course this has to be done carefully, e.g. checking the xpassing tests etc...

What is the difference between pytest.mark.xfail and pytest.xfail?

Thanks for figuring this out. Still, I think I have to test this locally - the time the CI takes is very inconsistent on azure.

Yes, I think this PR is helpful anyway and should bring down the ci time a bit.

pinning numba seems to have fixed the issue. It definitely is important to speed up our CI, though, waiting more than 30 minutes for the CI to finish is really not ideal.

I'm a bit confused, the windows CI used to take about as long as the macOS CI to complete. The last run for which that was true was about a week ago, does anyone know what changed since then?

Edit: maybe because of the release of numba=0.52.0 to conda-forge? If so, could you try pinning numba?

Here is what I learned: * The same tests are slow in windows and linux. Just, that those on windows take about twice as long. * The following test seem to be slow: * xarray/tests/test_distributed.py * many tests with sparse, e.g. xarray/tests/test_dataset.py::TestDataset::test_unstack_sparse * plotting tests, especially with FacetGrid

I am not sure what takes long in xarray/tests/test_distributed.py: writing the files or creating the cluster. If it is the latter, it could be possible to only open in once in the module (but I don't know if that actually works or if it has to be closed every time).

https://github.com/pydata/xarray/blob/76d5c0c075628475b555997b82c55dd18a34936e/xarray/tests/test_distributed.py#L118-L119

Windows py37

Linux py37

de-parametrize tests with a slow setup (if possible)

If I understand the proposal correctly — a different approach is to change the scope of the parameterizations rather than remove them — and then they only run once; e.g. scope="module"

No..., it failed at 99%. I don't entirely get it. The tests were well under way when I left. So I'd really be interested to get the timings of the tests to see what takes so long...

I just tried running the windows CI via Github actions

Yes, that's of course another good alternative.

other ideas?

I just tried running the windows CI via Github actions, and I noticed some improvements. The entire run takes ~ 45 minutes

If interested, here's the workflow configuration file I am using... Also, I am happy to submit a PR if need be.