I changed the title of this issue to reflect the current situation. PyNIO is unlikely to have another release and its compatibility with the Xarray and the rest of the ecosystem is quickly waning. If someone decides to pick up PyNIO maintenance in the future, they can provide backend entrypoint.

Does it make sense to move it to its own repo, seeing that this package is no longer maintained?

I suggest we plan to remove the pynio backend in a future release. The project is very likely dead.

https://github.com/NCAR/pynio/issues/53

My bad, I did not understand correctly what min-all-deps is for. I really have no clue if this can be solved, I only know that to get pynio to work with newer hdf5 required a change in build config for hdf5 (a compatibility macro) and a compiler flag in pynio.

Yes it works now but not after I updated dask and scipy versions in #6559

6559 is dropping PyNIO because I can't create a min-all-deps environment otherwise. We could try adding it back in one of the other environments.

I've downloaded a copy of min-all-deps.yml from this repo and run micromamba create -f min-all-deps.yml in a fedora container and it looks fine: ``` (xarray-tests) [root@f5f875d59f07 ~]# micromamba list List of packages in environment: "/root/micromamba/envs/xarray-tests"

Name Version Build Channel ──────────────────────────────────────────────────────────────────────────── _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge affine 2.3.1 pyhd8ed1ab_0 conda-forge antlr-python-runtime 4.7.2 py38h578d9bd_1003 conda-forge asciitree 0.3.3 py_2 conda-forge attrs 21.4.0 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.1 pyha770c72_0 conda-forge bokeh 2.3.3 py38h578d9bd_0 conda-forge boost-cpp 1.72.0 h8e57a91_0 conda-forge boto3 1.13.26 pyh9f0ad1d_0 conda-forge botocore 1.16.26 pyh9f0ad1d_0 conda-forge bottleneck 1.3.4 py38h3ec907f_0 conda-forge brotlipy 0.7.0 py38h0a891b7_1004 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cairo 1.16.0 hcf35c78_1003 conda-forge cartopy 0.17.0 py38h9cf8511_1015 conda-forge cdat_info 8.2.1 pyhd8ed1ab_2 conda-forge cdms2 3.1.5 py38h11a58eb_3 conda-forge cdtime 3.1.4 py38hac60b08_2 conda-forge certifi 2021.10.8 py38h578d9bd_2 conda-forge cf-units 3.0.1 py38h6c62de6_2 conda-forge cffi 1.15.0 py38h3931269_0 conda-forge cfgrib 0.9.8.5 pyhd8ed1ab_0 conda-forge cfitsio 3.470 h2e3daa1_7 conda-forge cftime 1.2.1 py38hab2c0dc_1 conda-forge charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge coverage 6.3.2 py38h0a891b7_2 conda-forge coveralls 3.3.1 pyhd8ed1ab_0 conda-forge cryptography 36.0.2 py38h2b5fc30_1 conda-forge curl 7.76.1 h979ede3_1 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.11.2 py38h0a891b7_2 conda-forge dask 2.30.0 py_0 conda-forge dask-core 2.30.0 py_0 conda-forge distarray 2.12.2 pyhd8ed1ab_2 conda-forge distributed 2.30.1 py38h578d9bd_0 conda-forge docopt 0.6.2 py_1 conda-forge docutils 0.15.2 py38h578d9bd_3 conda-forge eccodes 2.17.0 h59f7be3_1 conda-forge esmf 8.0.0 nompi_hb0fcdcb_6 conda-forge esmpy 8.0.0 nompi_py38hf0e99fa_1 conda-forge execnet 1.9.0 pyhd8ed1ab_0 conda-forge expat 2.4.8 h27087fc_0 conda-forge fasteners 0.17.3 pyhd8ed1ab_0 conda-forge fontconfig 2.14.0 h8e229c2_0 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.6 h7f98852_0 conda-forge fsspec 2022.3.0 pyhd8ed1ab_0 conda-forge future 0.18.2 py38h578d9bd_5 conda-forge g2clib 1.6.0 hf3f1b0b_9 conda-forge geos 3.8.1 he1b5a44_0 conda-forge geotiff 1.5.1 h05acad5_10 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.70.2 h780b84a_4 conda-forge glib-tools 2.70.2 h780b84a_4 conda-forge h5netcdf 0.8.1 py_0 conda-forge h5py 2.10.0 nompi_py38h513d04c_102 conda-forge hdf4 4.2.15 h10796ff_3 conda-forge hdf5 1.10.5 nompi_h7c3c948_1111 conda-forge hdfeos2 2.20 hca1e490_1001 conda-forge hdfeos5 5.1.16 h8b6279f_6 conda-forge heapdict 1.0.1 py_0 conda-forge hypothesis 6.46.1 pyhd8ed1ab_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 3.3 pyhd8ed1ab_0 conda-forge importlib-metadata 4.11.3 py38h578d9bd_1 conda-forge importlib_metadata 4.11.3 hd8ed1ab_1 conda-forge importlib_resources 5.7.1 pyhd8ed1ab_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge iris 2.4.0 py38_0 conda-forge jasper 1.900.1 h07fcdf6_1006 conda-forge jinja2 3.1.1 pyhd8ed1ab_0 conda-forge jmespath 0.10.0 pyh9f0ad1d_0 conda-forge jpeg 9e h166bdaf_1 conda-forge json-c 0.13.1 hbfbb72e_1002 conda-forge jsonschema 4.4.0 pyhd8ed1ab_0 conda-forge jupyter_core 4.9.2 py38h578d9bd_0 conda-forge kealib 1.4.13 hec59c27_0 conda-forge kiwisolver 1.4.2 py38h43d8883_1 conda-forge krb5 1.17.2 h926e7f8_0 conda-forge lazy-object-proxy 1.7.1 py38h0a891b7_1 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge libaec 1.0.6 h9c3ff4c_0 conda-forge libblas 3.8.0 17_openblas conda-forge libcblas 3.8.0 17_openblas conda-forge libcdms 3.1.2 hf60d256_111 conda-forge libcf 1.0.3 py38h88b7cc0_109 conda-forge libcurl 7.76.1 hc4aaa36_1 conda-forge libdap4 3.20.6 hd7c4107_2 conda-forge libdrs 3.1.2 hc2e2db3_113 conda-forge libdrs_f 3.1.2 hae7e664_111 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 11.2.0 h1d223b6_16 conda-forge libgdal 3.0.4 h3dfc09a_6 conda-forge libgfortran-ng 7.5.0 h14aa051_20 conda-forge libgfortran4 7.5.0 h14aa051_20 conda-forge libglib 2.70.2 h174f98d_4 conda-forge libgomp 11.2.0 h1d223b6_16 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 hd79254b_1012 conda-forge liblapack 3.8.0 17_openblas conda-forge libllvm10 10.0.1 he513fc3_3 conda-forge libnetcdf 4.7.4 nompi_h9f9fd6a_101 conda-forge libnghttp2 1.47.0 h727a467_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libopenblas 0.3.10 pthreads_hb3c22a3_5 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 12.3 h255efa7_3 conda-forge libspatialite 4.3.0a h2482549_1038 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_16 conda-forge libtiff 4.2.0 hdc55705_0 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp-base 1.2.2 h7f98852_1 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxml2 2.9.10 hee79883_0 conda-forge libxslt 1.1.33 h31b3aaa_0 conda-forge libzlib 1.2.11 h166bdaf_1014 conda-forge llvmlite 0.34.0 py38h4f45e52_2 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lxml 4.6.3 py38hf1fe3a4_0 conda-forge lz4-c 1.9.2 he1b5a44_3 conda-forge markupsafe 2.1.1 py38h0a891b7_1 conda-forge matplotlib-base 3.3.4 py38h0efea84_0 conda-forge msgpack-python 1.0.3 py38h43d8883_1 conda-forge nbformat 5.3.0 pyhd8ed1ab_0 conda-forge nc-time-axis 1.2.0 pyhd8ed1ab_2 conda-forge ncurses 6.3 h27087fc_1 conda-forge netcdf-fortran 4.5.2 nompi_h45d7149_104 conda-forge netcdf4 1.5.3 nompi_py38heb6102f_103 conda-forge numba 0.51.2 py38hc5bc63f_0 conda-forge numcodecs 0.9.1 py38h709712a_2 conda-forge numpy 1.18.5 py38h8854b6b_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openblas 0.3.10 pthreads_h43bd3aa_5 conda-forge openjpeg 2.3.1 hf7af979_3 conda-forge openssl 1.1.1n h166bdaf_0 conda-forge owslib 0.25.0 pyhd8ed1ab_0 conda-forge packaging 20.0 py_0 conda-forge pandas 1.1.5 py38h51da96c_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pillow 8.1.2 py38ha0e1e83_0 conda-forge pint 0.16.1 py_0 conda-forge pip 22.0.4 pyhd8ed1ab_0 conda-forge pixman 0.38.0 h516909a_1003 conda-forge pluggy 1.0.0 py38h578d9bd_3 conda-forge poppler 0.67.0 h14e79db_8 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 12.3 hc2f5b80_3 conda-forge proj 7.0.0 h966b41f_5 conda-forge pseudonetcdf 3.1.0 py_1 conda-forge psutil 5.9.0 py38h0a891b7_1 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge py 1.11.0 pyh6c4a22f_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pydap 3.2.2 pyh9f0ad1d_1001 conda-forge pyepsg 0.4.0 py_0 conda-forge pykdtree 1.3.4 py38h6c62de6_2 conda-forge pyke 1.1.1 pyhd8ed1ab_1004 conda-forge pynio 1.5.5 py38h031d99c_12 conda-forge pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.8 pyhd8ed1ab_0 conda-forge pyproj 2.6.1.post1 py38h7521cb9_0 conda-forge pyrsistent 0.18.1 py38h0a891b7_1 conda-forge pyshp 2.3.0 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 py38h578d9bd_5 conda-forge pytest 7.1.2 py38h578d9bd_0 conda-forge pytest-cov 3.0.0 pyhd8ed1ab_0 conda-forge pytest-env 0.6.2 py_0 conda-forge pytest-forked 1.4.0 pyhd8ed1ab_0 conda-forge pytest-xdist 2.5.0 pyhd8ed1ab_0 conda-forge python 3.8.13 h582c2e5_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.15.3 pyhd8ed1ab_0 conda-forge python_abi 3.8 2_cp38 conda-forge pytz 2022.1 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py38h0a891b7_4 conda-forge rasterio 1.1.5 py38h033e0f6_1 conda-forge readline 8.1 h46c0cb4_0 conda-forge requests 2.27.1 pyhd8ed1ab_0 conda-forge s3transfer 0.3.7 pyhd8ed1ab_0 conda-forge scipy 1.5.3 py38h828c644_0 conda-forge seaborn 0.11.2 hd8ed1ab_0 conda-forge seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge setuptools 59.8.0 py38h578d9bd_1 conda-forge shapely 1.7.1 py38ha11d057_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snuggs 1.4.7 py_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.1 pyhd8ed1ab_0 conda-forge sparse 0.11.2 py_0 conda-forge sqlite 3.38.3 h4ff8645_0 conda-forge statsmodels 0.13.2 py38h6c62de6_0 conda-forge tbb 2020.2 h4bd325d_4 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge tiledb 1.7.7 h8efa9f0_3 conda-forge tk 8.6.12 h27826a3_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge tornado 6.1 py38h0a891b7_3 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge typing_extensions 3.7.4.3 py_0 conda-forge tzcode 2022a h166bdaf_0 conda-forge udunits2 2.2.28 hc3e0081_0 conda-forge urllib3 1.25.11 py_0 conda-forge webob 1.8.7 pyhd8ed1ab_0 conda-forge wheel 0.37.1 pyhd8ed1ab_0 conda-forge xarray 2022.3.0 pyhd8ed1ab_0 conda-forge xerces-c 3.2.2 h8412b87_1004 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h7f98852_2 conda-forge zarr 2.5.0 py_0 conda-forge zict 2.2.0 pyhd8ed1ab_0 conda-forge zipp 3.8.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h166bdaf_1014 conda-forge zstd 1.4.8 hdf46e1d_0 conda-forge ```

6559 is dropping PyNIO because I can't create a min-all-deps environment otherwise. We could try adding it back in one of the other environments.

Reading here https://www.pyngl.ucar.edu/Nio.shtml it says that PyNIO is in "maintenance mode", what that means is not clear to me. There are no commits since 2019 to the the pynio repo https://github.com/NCAR/pynio.

We can uncomment here:

https://github.com/pydata/xarray/blob/305533d585389f7240ae2383a323337d4761d33a/ci/requirements/environment.yml#L36

which adds it back to the test suite. As the version does not change, the new build should also be picked up from

https://github.com/pydata/xarray/blob/305533d585389f7240ae2383a323337d4761d33a/ci/requirements/py38-min-all-deps.yml#L42

(it's nice to see that the build for pynio is fixed, but the package itself does not seem to be actively maintained).

Hello, sorry to bother you here, can you test again with newer pynio builds? I think pynio-feedstock#90 is now solved.

I would probably call it "xarray-pynio" (?) and move it to the "xarray-contrib" organisation.

I think this is OK. The README should say that it is totally unmaintained.

cc @khallock for awareness.

I agree, that would probably be the best place to put it. We might want do the same for pydap (which also seems to be unmaintained), too.

That would be nice. I would probably call it "xarray-pynio" (?) and move it to the "xarray-contrib" organisation. However, I think this needs a higher-level decision.

If you want I can take care to move pynio to an external repository, but we should decide where.

We just bumped the the minimum version of python-netCDF4 to 1.5 - we won't be able to bump it to 1.6 because of this. So we'll need to decide what to do with pynio before and properly deprecate it.

Worth noting though, that there is no python-netCDF4 version 1.6 at the moment, so this is not all that urgent. netCDF4 has not released a minor version in a long time (March 2019) (but they have done 12 patch releases).

4492 is merged. I changed the title to reflect that pynio should eventually be moved to an external repo but keep the issue open to have the discussion in place. I hope that this should not be too hard after the backend refactor.

Would it be enough to only test using py36-min-all-deps and add a warning to the docs stating that pynio conflicts with netcdf4>1.5.3, and maybe also that the pynio backend will be deprecated soon? Same for pydap, I guess.

Yes, fine for me - check #4492

Would it be enough to only test using py36-min-all-deps and add a warning to the docs stating that pynio conflicts with netcdf4>1.5.3, and maybe also that the pynio backend will be deprecated soon? Same for pydap, I guess.

This sounds fine to me.

not sure if we should repurpose py38-all-but-dask for that, right now the only difference between it and py38 is that dask (and all libraries that depend on it) are missing, which is intentional.

Would it be enough to only test using py36-min-all-deps and add a warning to the docs stating that pynio conflicts with netcdf4>1.5.3, and maybe also that the pynio backend will be deprecated soon? Same for pydap, I guess.

In py36-min-all-deps.yml we already test the newest version of pynio (1.5.5). My suggestion is therefore:

Keep pynio in - py36-min-all-deps.yml (pinned) - py38-all-but-dask.yml (not pinned to make sure we don't miss anything)

Remove pynio from: - py36.yml - py37.yml - py38.yml

Do you agree?

And long term what to do with the pynio backend? Deprecate? Move to an external repo?

Yes, in the long term I think the pynio backend will need to be moved outside of xarray, along with backends for other unmaintained projects such as pydap. This is one of goals of the backend refactor effort lead by @alexamici and @aurghs.

Until that happens, if new versions of netCDF4 and pynio are incompatible, my preferred work-around would be to reduce test coverage for pynio. It's not being developed actively, so there aren't strong reasons to maintain test coverage with every version of Python.