issues

As discussed in #6174, closing and re-opening a file between calls of to_netcdf can be avoided by using compute=False. In case of many write operations on the same file (e.g. for many separate groups in a netCDF4 file), this often improves the performance significantly. This PR makes note of this effect in the docstrings so that users can learn about it without diving into the source code.

• Unfortunately, it doesn't close #6174, because that issue also discusses the problem of obtaining a full list of all groups contained within a netCDF4 file when reading from that file.
• Changes to the doc strings are not mentioned in whats-new.rst, are they?

• [x] Closes #6310
• [x] Tests added
• [x] Doesn't include user visible changes or notable bug fixes for whats-new.rst
• [x] No new functions/methods for api.rst

As discussed in #6310, when setting "coordinates" as a variable attribute instead of as an encoding, the value of that variable attribute is overwritten when writing to a NetCDF file. This PR fixes it.

What happened?

When saving a dataset to a NetCDF file with both dimension and auxiliary coordinates, the data variables' coordinate attribute only lists the coordinates that are not related to any dimension (so-called "auxiliary" coordinates).

What did you expect to happen?

I expect xarray to adhere to the CF conventions where it says:

Every element of every feature must be unambiguously associated with its space and time coordinates and with the feature that contains it. The coordinates attribute must be attached to every data variable to indicate the spatiotemporal coordinate variables that are needed to geo-locate the data. [from https://cfconventions.org/cf-conventions/cf-conventions.html#coordinates-metadata]

See also this example: https://cfconventions.org/cf-conventions/cf-conventions.html#_single_trajectory

Minimal Complete Verifiable Example

Python import xarray as xr xr.Dataset({"values": ('time', [0.0, 0.1])}, coords={ 'time': ('time', [0, 1]), 'lat': ('time', [5, 4]), 'lon': ('time', [10, 12]) }).to_netcdf("test.nc")

bash $ ncdump -h test.nc netcdf test { dimensions: time = 2 ; variables: double values(time) ; values:_FillValue = NaN ; values:coordinates = "lon lat" ; int64 time(time) ; int64 lat(time) ; int64 lon(time) ; } Note that values:coordinates doesn't list the time coordinate (as in the example in the CF conventions).

Relevant log output

No response

Anything else we need to know?

I don't want to offend anyone, please correct me if I read the CF conventions in the wrong way or if this is a special case, and in general, the CF conventions recommend to only include auxiliary coordinates in the coordinates attribute. I couldn't find any such recommendation in the conventions text, but it's a very long document. What I did find (see my second post) is that it is "permissible, but optional" to list non-auxiliary variables in general, and it is required for the special case of "Discrete Sampling Geometries".

Environment

``` INSTALLED VERSIONS

commit: None python: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51) [GCC 9.4.0] python-bits: 64 OS: Linux OS-release: 5.4.0-100-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.1 libnetcdf: 4.8.1

xarray: 0.19.0 pandas: 1.3.4 numpy: 1.21.4 scipy: 1.7.0 netCDF4: 1.5.8 pydap: None h5netcdf: None h5py: 3.4.0 Nio: None zarr: None cftime: 1.5.1.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: 0.9.9.0 iris: None bottleneck: 1.3.2 dask: 2021.07.1 distributed: 2021.07.1 matplotlib: 3.4.2 cartopy: 0.20.1 seaborn: None numbagg: None pint: 0.17 setuptools: 59.2.0 pip: 21.2.1 conda: None pytest: None IPython: 7.29.0 sphinx: None ```

Is your feature request related to a problem?

I know that there is a big discussion going on in #4118 about organizing hierarchies of datasets within xarray's data structures. But this issue is supposed to address only a comparably simple aspect of this.

Suppose that you have a list ds_list of xarray.Dataset objects with different dimensions etc. and you want to store them all in one NetCDF4 file by using the group feature introduced in NetCDF4. The group name of each dataset is stored in ds_names. Obviously, you can do something like this: python for name, ds in zip(ds_names, ds_list): ds.to_netcdf(path, group=name) However, this is really slow when you have many (hundreds or thousands of) small datasets because the file is opened and closed in every iteration.

Describe the solution you'd like

I would like to have a function xr.to_netcdf that writes a list (or a dictionary) of datasets to a single NetCDF4 file with a single open/close operation. Ideally there should also be a way to read many datasets at once from a single NetCDF4 file using xr.open_dataset.

Describe alternatives you've considered

Currently, I'm using the following read/write functions to achieve the same:

```python import pathlib from xarray.backends import NetCDF4DataStore from xarray.backends.api import dump_to_store from xarray.backends.common import ArrayWriter from xarray.backends.store import StoreBackendEntrypoint

def _xr_to_netcdf_multi(path, ds_dict, encoding=None): """Write multiple xarray Datasets to separate groups in a single NetCDF4 file

Parameters
----------
path : str or Path
    Path of the target NetCDF file.
ds_dict : dict whose keys are group names and values are xr.Dataset
    Each xr.Dataset in the dict is stored in the group identified by its key in the dict.
    Note that an empty string ("") is a valid group name and refers to the root group.
encoding : dict whose keys are group names and values are dict, optional
    For each dataset/group, one dict that is compliant with the format of the `encoding`
    keyword parameter in `xr.Dataset.to_netcdf`. Default: None
"""
path = str(pathlib.Path(path).expanduser().absolute())
store = NetCDF4DataStore.open(path, "w", "NETCDF4", None)
try:
    writer = ArrayWriter()
    for group, dataset in ds_dict.items():
        store._group = group
        unlimited_dims = dataset.encoding.get("unlimited_dims", None)
        encoding = None if encoding is None or group not in encoding else encoding[group]
        dump_to_store(dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims)
finally:
    store.close()

def _xr_open_dataset_multi(path, prefix=""): """Read multiple xarray Datasets from groups contained in a single NetCDF4 file

Warning: The data is loaded into memory!

Parameters
----------
path : str or Path
    Path of the NetCDF file to read.
prefix : str, optional
    If given, only read groups whose name starts with this prefix. Default: ""

Returns
-------
ds_dict : dict whose keys are group names and values are xr.Dataset
    Each xr.Dataset in the dict is taken from the group identified by its key in the dict.
    Note that an empty string ("") is a valid group name and refers to the root group.
"""
path = str(pathlib.Path(path).expanduser().absolute())
store = NetCDF4DataStore.open(path, "r", "NETCDF4", None)
ds_dict = {}
try:
    groups = [g for g in _xr_nc4_groups_from_store(store) if g.startswith(prefix)]
    store_entrypoint = StoreBackendEntrypoint()
    for group in groups:
        store._group = group
        ds = store_entrypoint.open_dataset(store)
        ds.load()
        ds_dict[group] = ds
finally:
    store.close()
return ds_dict

def _xr_nc4_groups_from_store(store): """List all groups contained in the given NetCDF4 data store

Parameters
----------
store : xarray.backend.NetCDF4DataStore

Returns
-------
list of str
"""
def iter_groups(ds, prefix=""):
    groups = [""]
    for group_name, group_ds in ds.groups.items():
        groups.extend([f"{prefix}{group_name}{subgroup}"
                       for subgroup in iter_groups(group_ds, prefix="/")])
    return groups
with store._manager.acquire_context(False) as root:
    return iter_groups(root)

```

Additional context

No response

Is your feature request related to a problem?

There is this veeeeery old PR from 2016 about adding gzip support to NetCDF4 when used with OpenDAP: https://github.com/pydata/xarray/pull/817 The last comment, from 2018, claimed that "netCDF4-Python does not support Python file objects". However, this limitation has been solved a long time ago in 2017 in NetCDF4 (https://github.com/Unidata/netcdf4-python/pull/652).

Describe the solution you'd like

It would be great if xr.Dataset.to_netcdf as well as xr.open_dataset could add support for file-like objects when used with engine="netcdf4". Once this is done, it should be easy to also support .nc.gz files in case of the NETCDF4 format (which is currently only supported in case of NETCDF3).

Describe alternatives you've considered

No response

Additional context

No response