issue_comments

I think this was fixed by https://github.com/Unidata/netcdf-c/issues/2573

Building against netcdf with the patch backported by @opoplawski, I now get a single failure: ``` ____ TestNCZarr.testopen_nczarr ______ [gw3] linux -- Python 3.11.1 /usr/bin/python3 zarr_obj = <zarr.core.Array '/dim2' (9,) float64 read-only> dimension_key = '_ARRAY_DIMENSIONS', try_nczarr = True def _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr): # Zarr arrays do not have dimensions. To get around this problem, we add # an attribute that specifies the dimension. We have to hide this attribute # when we send the attributes to the user. # zarr_obj can be either a zarr group or zarr array try: # Xarray-Zarr

      dimensions = zarr_obj.attrs[dimension_key]

/builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:183:

self = <zarr.attrs.Attributes object at 0x7f697d934ad0> item = '_ARRAY_DIMENSIONS' def getitem(self, item):

  return self.asdict()[item]

E KeyError: '_ARRAY_DIMENSIONS' /usr/lib/python3.11/site-packages/zarr/attrs.py:74: KeyError During handling of the above exception, another exception occurred: zarr_obj = <zarr.core.Array '/dim2' (9,) float64 read-only> dimension_key = '_ARRAY_DIMENSIONS', try_nczarr = True def _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr): # Zarr arrays do not have dimensions. To get around this problem, we add # an attribute that specifies the dimension. We have to hide this attribute # when we send the attributes to the user. # zarr_obj can be either a zarr group or zarr array try: # Xarray-Zarr dimensions = zarr_obj.attrs[dimension_key] except KeyError as e: if not try_nczarr: raise KeyError( f"Zarr object is missing the attribute {dimension_key}, which is " "required for xarray to determine variable dimensions." ) from e

        # NCZarr defines dimensions through metadata in .zarray
        zarray_path = os.path.join(zarr_obj.path, ".zarray")
        zarray = json.loads(zarr_obj.store[zarray_path])
        try:
            # NCZarr uses Fully Qualified Names
            dimensions = [

              os.path.basename(dim) for dim in zarray["_NCZARR_ARRAY"]["dimrefs"]
            ]

E KeyError: '_NCZARR_ARRAY' /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:197: KeyError The above exception was the direct cause of the following exception: self = <xarray.tests.test_backends.TestNCZarr object at 0x7f697ed62f10> def test_open_nczarr(self) -> None: with create_tmp_file(suffix=".zarr") as tmp: expected = self._create_nczarr(tmp) actual = xr.open_zarr(tmp, consolidated=False) /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/tests/test_backends.py:5741:

/builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:819: in open_zarr ds = open_dataset( /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/api.py:540: in open_dataset backend_ds = backend.open_dataset( /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:897: in open_dataset ds = store_entrypoint.open_dataset( /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/store.py:28: in open_dataset vars, attrs = store.load() /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/common.py:128: in load (_decode_variable_name(k), v) for k, v in self.get_variables().items() /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:480: in get_variables return FrozenDict( /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/core/utils.py:469: in FrozenDict return Frozen(dict(args, *kwargs)) /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:481: in <genexpr> (k, self.open_store_variable(k, v)) for k, v in self.zarr_group.arrays() /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:457: in open_store_variable dimensions, attributes = _get_zarr_dims_and_attrs(

zarr_obj = <zarr.core.Array '/dim2' (9,) float64 read-only> dimension_key = '_ARRAY_DIMENSIONS', try_nczarr = True def _get_zarr_dims_and_attrs(zarr_obj, dimension_key, try_nczarr): # Zarr arrays do not have dimensions. To get around this problem, we add # an attribute that specifies the dimension. We have to hide this attribute # when we send the attributes to the user. # zarr_obj can be either a zarr group or zarr array try: # Xarray-Zarr dimensions = zarr_obj.attrs[dimension_key] except KeyError as e: if not try_nczarr: raise KeyError( f"Zarr object is missing the attribute {dimension_key}, which is " "required for xarray to determine variable dimensions." ) from e

        # NCZarr defines dimensions through metadata in .zarray
        zarray_path = os.path.join(zarr_obj.path, ".zarray")
        zarray = json.loads(zarr_obj.store[zarray_path])
        try:
            # NCZarr uses Fully Qualified Names
            dimensions = [
                os.path.basename(dim) for dim in zarray["_NCZARR_ARRAY"]["dimrefs"]
            ]
        except KeyError as e:

          raise KeyError(
                f"Zarr object is missing the attribute `{dimension_key}` and the NCZarr metadata, "
                "which are required for xarray to determine variable dimensions."
            ) from e

E KeyError: 'Zarr object is missing the attribute _ARRAY_DIMENSIONS and the NCZarr metadata, which are required for xarray to determine variable dimensions.' /builddir/build/BUILDROOT/python-xarray-2022.12.0-1.fc38.x86_64/usr/lib/python3.11/site-packages/xarray/backends/zarr.py:200: KeyError ```

Does that look like a problem in netcdf or a problem in xarray here?

I could, but I'm not sure what to try. I tried running https://github.com/Unidata/netcdf4-python/blob/master/examples/tutorial.py with those versions and that worked without issue.

Still broken in 2022.11.0; any ideas what to look at here?

IIUC, I think this is the exception those tests are trying to check: https://github.com/pydata/xarray/blob/4880012ddee9e43e3e18e95551876e9c182feafb/xarray/plot/utils.py#L893-L896

We looked at this downstream in Matplotlib to see if we introduced something in 3.6.0rc2. There was a bug there in that exception types changed unexpectedly.

However, I am thinking that these two tests are incorrect. They are generically checking for ValueError without a match on the exception text. This means they completely missed the fact that husl was an invalid colormap (which was only exposed because 3.6.0rc2 accidentally changed the type for that exception), whereas I think it was only supposed to check that levels was not specified.

Backporting netcdf 4.9.0 to Fedora 36 also OOMs, so there's something weird there, but not sure if it's netCDF, or something that xarray is doing.

Hmm, I seem to be able to build on Fedora 36 (instead of Rawhide above), with the following versions:

Have you tried with the latest Cartopy?

What version of bottleneck are you testing it with? We require bottleneck >=1.3 (https://github.com/pydata/xarray/blob/master/ci/requirements/py37-min-all-deps.yml)

This should go in setup.cfg, then? I'm not using conda.

Ah, you're right, it's still 1.2.1, I'll fix that.

That seems insufficient: ```pytb _____ testpush_dask ______ [gw3] linux -- Python 3.9.5 /usr/bin/python3 @requires_dask @requires_bottleneck def test_push_dask(): import bottleneck import dask.array

    array = np.array([np.nan, np.nan, np.nan, 1, 2, 3, np.nan, np.nan, 4, 5, np.nan, 6])
    expected = bottleneck.push(array, axis=0)
    for c in range(1, 11):
        with raise_if_dask_computes():
            actual = push(dask.array.from_array(array, chunks=c), axis=0, n=None)

      np.testing.assert_equal(actual, expected)

/builddir/build/BUILDROOT/python-xarray-0.18.2-1.fc35.x86_64/usr/lib/python3.9/site-packages/xarray/tests/test_duck_array_ops.py:887:

/usr/lib/python3.9/site-packages/dask/array/core.py:1476: in array x = self.compute() /usr/lib/python3.9/site-packages/dask/base.py:285: in compute (result,) = compute(self, traverse=False, kwargs) /usr/lib/python3.9/site-packages/dask/base.py:567: in compute results = schedule(dsk, keys, kwargs) /usr/lib/python3.9/site-packages/dask/local.py:560: in get_sync return get_async( /usr/lib/python3.9/site-packages/dask/local.py:503: in get_async for key, res_info, failed in queue_get(queue).result(): /usr/lib64/python3.9/concurrent/futures/_base.py:438: in result return self.get_result() /usr/lib64/python3.9/concurrent/futures/_base.py:390: in __get_result raise self._exception /usr/lib/python3.9/site-packages/dask/local.py:545: in submit fut.set_result(fn(args, kwargs)) /usr/lib/python3.9/site-packages/dask/local.py:237: in batch_execute_tasks return [execute_task(a) for a in it] /usr/lib/python3.9/site-packages/dask/local.py:237: in <listcomp> return [execute_task(a) for a in it] /usr/lib/python3.9/site-packages/dask/local.py:228: in execute_task result = pack_exception(e, dumps) /usr/lib/python3.9/site-packages/dask/local.py:223: in execute_task result = _execute_task(task, data) /usr/lib/python3.9/site-packages/dask/core.py:121: in _execute_task return func((_execute_task(a, cache) for a in args)) /usr/lib/python3.9/site-packages/dask/optimization.py:963: in __call return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args))) /usr/lib/python3.9/site-packages/dask/core.py:151: in get result = _execute_task(task, cache) /usr/lib/python3.9/site-packages/dask/core.py:121: in _execute_task return func(*(_execute_task(a, cache) for a in args))

func = <built-in function push>, args = [array([nan, 1.])] kwargs = {'axis': 0, 'n': None} def apply(func, args, kwargs=None): if kwargs:

      return func(*args, **kwargs)

E TypeError: n must be an integer /usr/lib/python3.9/site-packages/dask/utils.py:34: TypeError ```

It's not migrated yet; the 3.4.0rc1 prerelease is on PyPI now.

FAILED xarray/tests/test_plot.py::TestContour::test_single_level - ValueError... FAILED xarray/tests/test_plot.py::test_facetgrid_single_contour - ValueError:...

This comes from https://github.com/matplotlib/matplotlib/pull/17830, as BoundaryNorm/contour needs at least two levels, but the test is explicitly using one. If that's something you think is needed, please report the use case to Matplotlib.

@pep8speaks seems a little broken... Line 328 is an empty line.

Done.

Sorry I lost track of this; I will rebase soon and update.

What does this mean? I couldn't find anything in the release notes...

https://matplotlib.org/api/api_changes.html#passing-a-line2d-s-drawstyle-together-with-the-linestyle-is-deprecated

• First one: https://github.com/DanielStutzbach/heapdict/pull/13
• Second/third one: https://github.com/pydap/pydap/issues/203
• linestyle/drawstyle one: #3274

If slow tests are being run by default, I'm not sure they really need their own special option. You can mark them (with @pytest.mark.slow) and use pytest's builtin selectors to not run them: pytest -m "not slow".

Are flaky tests actually flaky or do they just not work? If flaky, and a re-run will help, then maybe try the pytest-rerunfailures plugin.

See #1298, where I changed Python 2.7 CI to install h5netcdf as well (it seemed to be missing.) This test doesn't work on Python 2 either right now.

Oh, just realized there's a skip on Python 3, which links to #535. Apparently, this test is now broken on both Python versions.

shoyer/h5netcdf#16 appears to be the upstream issue, but oddly, it seems to be reported against Python 3 when here it fails with Python 2.

Both netcdf4 and h5netcdf are installed now; that's the point.

Fortunately, this does fail in the same way as it does for me locally. Since Python 3 works, I think this might be a bytes/str thing, but the error is a bit opaque to me.

That does seem like a bug, but I'm not the one to ask. I'd file a bug and ping @efiring.