For some context, without this fix, there would be strange errors - not only for developers of xarray, but also for developers of third-party libraries which has a import xarray somewhere in the code.

See also #5236 which could be related which uses conda.

I tried to scour through pip and pytest issues, but I can't find a discussion on this. pip and pytest and pycacheare too generic keywords. I even found pytest plugin to remove bytecode but it does not work ~~anymore~~ for me.

I proposed a fix in #6039, which I believe is good to have, as you can't expect every user to execute with python -B or set PYTHONDONTWRITEBYTECODE while running tests.

This in my opinion is a cascade of multiple issues: - pytest creates special bytecode - pip does not clean the whole __pycache__ directory on uninstall - xarray.pycompat.is_duck_dask_array uses importlib.import_module('dask') to determine if it is installed.

@twhughes @keewis In my case, I encounter this when I: - Have xarray + dask installed in my environment - Run any pytest which imports xarray (which in turn imports dask) - Uninstall dask, but pytest artefacts remain:

sh ❯ ls venv/lib/python3.9/site-packages/dask/__pycache__ utils_test.cpython-39-pytest-6.2.5.pyc

Here are the steps which would hopefully reproduce the behaviour:

```py

Create this small unit test named:

test_import_xarray.py

def test_xr(): import xarray ```

```sh python -m venv venv source venv/bin/activate pip install xarray pytest dask pytest test_import_xarray.py

pip uninstall dask ls venv/lib/python3.9/site-packages/dask/pycache # See output above python -c 'import dask' # No ImportError! python -c 'import xarray' ```

Exact versions used Python 3.9.9 along with: ```sh

requirements.txt

attrs==21.2.0 cloudpickle==2.0.0 dask==2021.11.2 fsspec==2021.11.1 iniconfig==1.1.1 locket==0.2.1 numpy==1.21.4 packaging==21.3 pandas==1.3.4 partd==1.2.0 pluggy==1.0.0 py==1.11.0 pyparsing==3.0.6 pytest==6.2.5 python-dateutil==2.8.2 pytz==2021.3 PyYAML==6.0 six==1.16.0 toml==0.10.2 toolz==0.11.2 xarray==0.20.1 ```

To use the select method, the following should be changed:

• Swap the imports so that newer importlib_metadata is imported first: https://github.com/pydata/xarray/blob/e0deb9cf0a5cd5c9e3db033fd13f075added9c1e/xarray/backends/plugins.py#L8-L12
• Require importlib_metadata for Python < 3.10 https://github.com/pydata/xarray/blob/e0deb9cf0a5cd5c9e3db033fd13f075added9c1e/setup.cfg#L81

@kmuehlbauer

• Add a breakpoint() below line 102 and print out entrypoints. Step into build_engines follow it using the debugger.
• Can you also try this suggestion https://github.com/pydata/xarray/issues/5944#issuecomment-962414054

On a side note, the syntax .get is deprecated in the importlib_metadata package and most likely in Python 3.10's importlib.metadata stdlib.

```py

In [16]: from importlib_metadata import entry_points

In [17]: entry_points().get('xarray.backends', ()) <ipython-input-17-5f3ea0df5c10>:1: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select. entry_points().get('xarray.backends', ()) Out[17]: [EntryPoint(name='rasterio', value='rioxarray.xarray_plugin:RasterioBackend', group='xarray.backends'), EntryPoint(name='pymech', value='pymech.dataset:PymechXarrayBackend', group='xarray.backends')]

In [18]: entry_points().select(group='xarray.backends') Out[18]: [EntryPoint(name='rasterio', value='rioxarray.xarray_plugin:RasterioBackend', group='xarray.backends'), EntryPoint(name='pymech', value='pymech.dataset:PymechXarrayBackend', group='xarray.backends')] ```

Or add a small dummy package to be installed using pip while testing. Something like the following under the ci/requirements/*.yml files should do it.

yml - pip: - ./xarray_test_package

with the following contents at ci/requirements at the bare minimum ```ini ❯ cat xarray_test_package/pyproject.toml [build-system] requires = ["setuptools", "wheel"] build-backend = "setuptools.build_meta" ❯ cat xarray_test_package/setup.cfg [metadata] name = xarray_test_package version = 0.0.0

[options] packages = find:

[options.entry_points] xarray.backends = xarray_test_package_backend = xarray_test_package.plugin:XarrayTestPackageBackend ```

This can later be extended with a minimal xarray_test_package.plugin module if needed.

In the same example 1st party backends such as h5netcdf work with both versions.

The only hiccup that I noticed was scitools-iris requires cartopy (which would install only if numpy is pre-installed), which in turn requires cf-units (which again would install if UDUNITS-2 library is present). I had gdal installed in my machine so I did not notice this. In short, it is best to avoid / mock scitools+ stuff if it can be avoided. I added it as a doc dependency because it is mandatory to have scitools+ packages - the way it is configured now. There is certainly room for future improvement.

Iris on PyPI is a dummy package. The package relevant for us is scitools-iris which can be difficult to install. That and some other packages should be installed to generate the docs. Ideally these should be mocked, maybe for another issue.

I tracked down all the dependencies required for docs. Could be useful for development.

@mathause Something similar to https://docs.dask.org/en/latest/install.html#pip ?

I suspect system jitter in the profiling as the time for Dataset.isel went up. It would be useful to run sudo python -m pyperf system tune before running profiler/benchmarks.

At first sight it looks somewhat like a hybrid between Cython (for the ahead-of-time transpiling to C++) and numba (for having python-compatible syntax).

Not really. Pythran always releases the GIL and does a bunch of optimizations between transpilation and compilations.

A good approach would be try out different compilers and see what performance is obtained, without losing readability (https://github.com/pydata/xarray/issues/2799#issuecomment-469444519). See scikit-image/scikit-image/issues/4199 where the package transonic was being experimentally tested to replace Cython-only code with python code + type hints. As a bonus, you get to switch between Cython, Pythran and Numba,