issue_comments

The Compatibility Note I linked above appears to point out the the entry_points(group="xarray.backends") is available since Python 3.6, so we not need the versione check at all.

There's some unfortunate version numbers there. That refers to version 3.6 of the third-party importlib_metadata library, a later version of which was used for the importlib.metadata library included with Python 3.10. If you try to run entry_points(group="xarray.backends") on Python 3.9 it fails with a TypeError.

Just came to report this after seeing it in some tests of one of my projects. Based on a change I made for plugins in that project, I think that changing

https://github.com/pydata/xarray/blob/d479009d79374dc4a56c9f4346b1af38f5ac182c/xarray/backends/plugins.py#L96-L99

to

python @functools.lru_cache(maxsize=1) def list_engines(): try: entrypoints = entry_points(group="xarray.backends") # Python >= 3.10 except TypeError: entrypoints = entry_points().get("xarray.backends", ()) return build_engines(entrypoints)

will take care of this. Not tested as I don't have access to an older Python environment right now, but I can make a pull request later if this would be a suitable workaround and nobody else does so first.

@aurghs it fixes it for both the test script in #5132 (using all three netCDF engines, netcdf4, h5netcdf and scipy) and for my original unit tests where I found the problem.

Added conversion of dimension names since non-string dimensions are intended to be supported. This will still fail for datasets as the dimension sorting does not yet support this.

I've changed the title.

I think improving the error message is the way to go.

Agreed. ~~Just playing with it, the attributes also need to be strings for netCDF (I'm not familiar with the other backends) so an equivalent error message should be added for them.~~ There is already a good error message for attributes, at least for netCDF, I wasn't looking at it properly.

This is also something that the backends will need to consider:

```python traceback

xr.DataArray(np.ones(5)).rename(dim_0=1).to_netcdf("/tmp/test.nc") ... /usr/lib/python3.9/site-packages/xarray/backends/netCDF4_.py in set_dimension(self, name, length, is_unlimited) 439 def set_dimension(self, name, length, is_unlimited=False): 440 dim_length = length if not is_unlimited else None --> 441 self.ds.createDimension(name, size=dim_length) 442 443 def set_attribute(self, key, value):

src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.createDimension()

src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dimension.init()

TypeError: expected bytes, int found ` I guess they could either convert to string (which will result in a different array when loaded) or a better error message (netCDF files only support strings for dimension names`` or similar).

Should I change the title here to be more general or would you prefer to close this and continue discussion in the other issue you linked?

I've created #5149 to fix this for attributes. Dimensions are inconsistent as to whether they are always strings or not, I've opened #5148 about this inconsistency.

LGTM. I don't know how we would test this...

For ensuring absolute paths, my MVCE in #5132 could be adapted.

Expansion of ~ might be trickier, but the os.path.expanduser docstring says

On Unix, an initial ~ is replaced by the environment variable HOME if it is set

and

On Windows, USERPROFILE will be used if set

so tests run with appropriate environment variables could be used with OS checking.

@aurghs & @dcherian note that at the moment most backends accept pathlib.Path (but it is not tested), so the isintance(..., str) doesn't run expandpath in that case.

What do you suggest?

For Python 3.6+ expanduser works with paths, returning a string:

```

os.path.expanduser(pathlib.Path("~/file.nc")) '/home/username/file.nc' ``` If its preferred to keep a Path instance, then the Path.expanduser method is the equivalent.

I think this would also fix #5132 if os.path.abspath was added around the os.path.expanduser calls.

@kmuehlbauer that doesn't as-is but is related. os.path.expanduser("test.nc") returns test.nc. Adding os.path.abspath around the expanduser call would probably do the trick. I'll comment over there.

Not for this PR but I wonder if xr.DataArray([1]).sum(min_count=2) should actually return NA.

It would make it more consistent, the current behaviour is

• xr.DataArray([1]).sum(min_count=2) -> 1
• xr.DataArray([1.0]).sum(min_count=2) -> nan

due to the different dtype defaults for skipna. I guess this would require rework of the dispatch logic in xarray.core.duck_array_ops._create_nan_agg_method to make it call nansum/nanprod when appropriate (e.g., should setting min_count imply skipna=True?).

• Changed to use duck_array_ops.where
• Docstring updated and entry added to breaking changes of whats new (I assumed this was appropriate). Note its a bit more nuanced than @mathause stated: for an integer array, skipna defaults to False and so xarray.core.duck_array_ops._create_nan_agg_method does not call nansum. This means you have to force skipna=True to see the difference:
  • master: xr.DataArray([1]).sum(min_count=1).dtype -> int64
  • this PR: xr.DataArray([1]).sum(min_count=1).dtype -> int64
  • master: xr.DataArray([1]).sum(skipna=True, min_count=1).dtype -> int64
  • this PR: xr.DataArray([1]).sum(skipna=True, min_count=1).dtype -> float64

Hopefully this is clear in the docstring changes, suggestions welcome.

@dcherian it looks like that works. A better test script:

```python import numpy as np import xarray as xr from xarray.tests import raise_if_dask_computes

def worker(da): if da.shape == (0, 0): return da

return da.where(da > 1)

np.random.seed(1023) da = xr.DataArray( np.random.normal(size=(20, 500)), dims=("x", "y"), coords=(np.arange(20), np.arange(500)), )

da = da.chunk(dict(x=5)) lazy = da.map_blocks(worker)

with raise_if_dask_computes(): result = lazy.sum("x", skipna=True, min_count=5)

result.load()

assert np.isnan(result[0]) assert not np.isnan(result[6]) ```

If I then remove the if null_mask.any() check and the following block, and replace it with

python dtype, fill_value = dtypes.maybe_promote(result.dtype) result = result.astype(dtype) result = np.where(null_mask, fill_value, result) it passes. I can start working on a pull request with these tests and changes if that looks acceptable to you.

~~How would you suggest handling the possible type promotion from the current dtype, fill_value = dtypes.maybe_promote(result.dtype) line? Currently it only tries promoting if the mask is True anywhere. Always promote, or just use the fill value and hope it works out?~~

A quick check with the debugger and it is the null_mask.any() call that is causing it to compute.

I think I've found another problem with _maybe_null_out if it is reducing over all dimensions. With the altered MCVE

```python import numpy as np import xarray as xr

def worker(da): if da.shape == (0, 0): return da

res = xr.full_like(da, np.nan)
res[0, 0] = 1
return res

da = xr.DataArray( np.random.normal(size=(20, 500)), dims=("x", "y"), coords=(np.arange(20), np.arange(500)), )

da = da.chunk(dict(x=5)) lazy = da.map_blocks(worker) result_allaxes = lazy.sum(skipna=True, min_count=5) result_allaxes.load() ```

I would expect result_allaxes to be nan since there are four blocks and therefore four non-nan values, less than min_count. Instead it is 4.

The problem seems to be the dtype check:

https://github.com/pydata/xarray/blob/5296ed18272a856d478fbbb3d3253205508d1c2d/xarray/core/nanops.py#L39

The test returns True for float64 and so the block isn't run. Another MCVE:

```python import numpy as np from xarray.core import dtypes

print(dtypes.NAT_TYPES) print(np.dtype("float64") in dtypes.NAT_TYPES) ```

Output: console (numpy.datetime64('NaT'), numpy.timedelta64('NaT')) True where I think False would be expected. Should I open a separate issue for this or can we track it here too?

grepping the code, the only other function that calls _maybe_null_out is prod, and I can confirm the problem also exists there. Updated the title, MCVE for prod:

```python import numpy as np import xarray as xr

def worker(da): if da.shape == (0, 0): return da

raise RuntimeError("I was evaluated")

da = xr.DataArray( np.random.normal(size=(20, 500)), dims=("x", "y"), coords=(np.arange(20), np.arange(500)), )

da = da.chunk(dict(x=5)) lazy = da.map_blocks(worker) result1 = lazy.prod("x", skipna=True) result2 = lazy.prod("x", skipna=True, min_count=5) ```