issue_comments

that appears to work on both my laptop and my local HPC, and is arguably a lot easier to implement / understand as we don't need to make sure all the globals we use are still available (which in this case would be acquire, OPTIONS, warnings, and RuntimeWarning).

Edit: let me change the PR to do that instead

Have you verfied that this fixes things at least on your machine?

I thought I did, but apparently something changed: right now it fails because OPTIONS is not available anymore, so we might have to add a reference to that, as well. Additionally, for the warning we use warnings.warn and RuntimeWarning that might also have disappeared already (but those are standard library / builtins, so hopefully not?)

In any case, you can verify this, too: - create a new environment using mamba create -n test python=3.11 numpy pandas packaging pooch netcdf4 and activate it - run pip install 'dask[array]' to install dask without distributed (this appears to make a difference for me, not sure if that's the same elsewhere) - editable-install xarray so we can easily switch between branches - run python -c 'import xarray as xr; ds = xr.tutorial.open_dataset("air_temperature", chunks={})'

This should print an error for main and shouldn't with this branch (confirmed on my local HPC).

the issue is that this doesn't occur on normal garbage collection but only on interpreter shutdown. So really, I don't think we have any way to test this using pytest as that itself is written in python (unless of course we can make use of sub-interpreters, but that might be more trouble than it's worth).

this is #7879 (and thus probably #7079). I suspect our locks are not working properly, but in any case we really should try to fix this.

No module named '_cffi_backend'

Does simply import cfgrib work for you? I suspect it doesn't, which would explain the issue. It's unfortunate that the error is rewritten to "unknown engine", but I'm not sure how we would detect that it's a dependency that fails.

No, this should still work: sh conda create -n test -c conda-forge xarray ipython python=3.11 cfgrib pooch conda activate test ipython ```python import xarray as xr

xr.tutorial.open_dataset("era5-2mt-2019-03-uk.grib") `` We somewhat recently dropped the builtincfgribengine in favor of the one provided by thecfgribpackage (and that is also the reason why the example in the docs fails:cfgrib` is not installed into the docs environment anymore, which is definitely an oversight).

Which version of cfgrib do you have? For reference, in the environment built with the command above I have cfgrib=0.9.10.3

great, thanks for the confirmation!

as stated by the exception, the cfgrib engine is unknown, which usually means you're missing the cfgrib package (or this is a environment issue). If you did indeed install it, can you post the output of either conda list or pip list (in case you're not using conda)?

I would have thought that the global (or module level here) variable/function aquire should have at least one reference until after the deletion of the object.

I think this is intended (though certainly not very easy to get right): see the second part of the warning in the __del__ documentation.

If I'm reading the different issues correctly, that means this is a duplicate of #7079

closing, since anything still missing should be feature requests for xncml

I think this has been fixed by xncml and/or kerchunk.

Actually, it goes away with pip install jinja2. We don't use jinja2 at all, so either this is some kind of weird effect on garbage collection (a timing issue?), or dask is doing something differently as soon as jinja2 is available.

Edit: most likely this is a timing issue... the offending line tries to make use of the internal acquire function, which I think at that point has already been destroyed. To fix that, I think we need to somehow store a reference on the file manager?

I can reproduce this locally: - download and unpack the files from https://github.com/pydata/xarray/issues/7814#issuecomment-1535168128 - use mamba create -n test python=3.11 xarray netcdf4 to create the environment (note: no dask) - use pip install "dask[array]" to install dask (does not pull distributed like the package from conda-forge) - put the code into a script and execute it

For reference, the full traceback is: pytb Exception ignored in: <function CachingFileManager.__del__ at 0x7fbdb237b430> Traceback (most recent call last): File "/home/jmagin/.local/opt/mambaforge/envs/test/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 246, in __del__ TypeError: 'NoneType' object is not callable

As far as I can tell, this means we're using something from distributed with a broken fallback, since the error goes away as soon as I install distributed.

how did you set up your environment? This works for me: sh mamba create -n test python=3.11 xarray dask netcdf4 pooch ipython mamba activate test ipython python xr.tutorial.open_dataset("rasm", chunks={})

Interestingly enough, though, is that you should only see this with xarray=2023.5.0, while your environment claims to have xarray=2023.4.2. It seems there is something wrong with your environment?

I agree, this change should be fine.

this appears to have worked. Thanks, @martinfleis!

add the token as a ANACONDA_NIGHTLY secret.

done. Let's try it!

Do we need to build the wheel in the same way you currently do in [the TestPyPI workflow]

depends on whether we can put local versions on anaconda. PyPI and TestPyPI don't allow that, so we had to modify pyproject.toml to make setuptools-scm generate versions without the local part.

Do you have an account there so I can add you?

now I do, the username is the same as on github.

since the version with cdms2 fails on environment creation we can't really remove the special case for python=3.11 until we reached a decision on what to do with cdms2 support. I'll open a new issue for that, but otherwise this should be ready for merging.

I wonder if pure-python projects could get away with asking to install from github? This works today: sh pip install git+https://github.com/pydata/xarray.git and we have been using it downstream in nightly CI.

That said, I can see a central location being helpful, and we'd certainly be happy to add a scheduled github action to upload built packages. We'd need some help with setting that up, though, as I personally don't have any experience whatsoever in uploading to anaconda.org. Do you have documentation/examples on how to do the actual upload?

That would also allow us to stop publishing builds to TestPyPI (not sure if those are nightlies), as that seems to have accumulated quite a few packages already that due to the PyPI policy stay there forever.

with #7855 and the recent change to pint we're finally down to just two test failures (and a whole lot of warnings): xarray/tests/test_dataarray.py::TestDataArray::test_to_and_from_cdms2_sgrid: ValueError: operands could not be broadcast together with shapes (3,) (4,) xarray/tests/test_ufuncs.py::test_unary: AssertionError: assert (<class 'int'> is <class 'numpy.float64'> or 1.0 == 0.9999999999999999) The first one looks like cdms2 is incompatible with a change in numpy>=1.25. Not sure if we can do anything about that, especially since there's a big warning on the cdms2 repo stating that the package is going to be retired / archived by the end of this year... I guess we should start thinking about removing cdms2 support?

The second looks like a precision issue, which we should be able to resolve by using assert_allclose instead... not sure, though, especially given numpy/numpy#23773. Otherwise we could just defer to whatever numpy is doing (of course, that assumes that full_like works properly, which might not be a good idea for a unit test): ```python @pytest.mark.parametrize( "a", [ xr.Variable(["x"], [0, 0]), xr.DataArray([0, 0], dims="x"), xr.Dataset({"y": ("x", [0, 0])}), ], ) def test_unary(a): fill_value = np.cos(0)

expected = xr.full_like(a, fill_value=fill_value, dtype=fill_value.dtype)
actual = np.cos(a)

assert_identical(actual, expected)

`` Edit: if relying onfull_like` turns out to be a concern, maybe we could use "copy + assign" instead?

let's have a look at what defines the object: python data = xr.DataArray(np.random.randn(2, 3), dims=("x", "y"), coords={"x": [10, 20]}) That means we have: - random data - two dimensions: x with a length of 2 and y with a length of 3 - one coordinate: x with values [10, 20]

The line python data.x.attrs["units"] = "x units" can be split into two parts: python x = data.x and python x.attrs["units"] = "x units" where x is the coordinate.

If we did the same thing with y we would not get the dimension, but a dummy coordinate that has the values np.arange(data.sizes["y"]). This is admittedly a bit confusing.

I agree, coordinate is the correct term. We could make that a bit less confusing by using a non-dimension coordinate in the example, but then again the point is that any coordinate can have additional metadata.

But I think the concept of dimension-coordinates (when the coordinate has the same name as the dimension) is still relevant.

If I remember correctly, while they lost their status as the only ones having an index dimension coordinates are still used for alignment.

pint<0.21 should work, and I'm looking into how this could be fixed, see hgrecco/pint#1660 and hgrecco/pint#1749. For the latter we might have to mark the "pint wrapping dask" test as requiring pint>=0.21 and make it use an explicit registry.

reopening so I can keep track of the second task: creating and automatically updating the support table

the first part of the policy change discussed in #7765 should be ready for review and merging. Still pondering how to implement the second part, so that will be a separate PR.

we still want to be able to explicitly switch it off as we did for bottleneck (mostly for debugging purposes), so the kwarg / option would be good to have either way.

In the meeting today we decided to both change the min-versions policy for python to 30 months, and to add a release table (something like the one in NEP29, but automatically created with each newly released version – possibly through a polling action that runs once a week or something so we don't increase the docs build time any more). The reason for the latter is that the policy, while a good policy, is not easy to understand, and its full meaning can only be inferred by reading the min_versions_policy script.

see also #7765, where there is a bit more information and discussion.

In short, the policy did work like NEP-29 before, but stopped doing so with python 3.8 because python switched from a irregular ~18 month release cycle to a regular 12 month cycle. We can adjust our policy by extending these 24 months to 30 months, which makes it align with NEP-29's 42 months window.

that's a numpy array with sparse data. What @TomNicholas was talking about is a array of type sparse.COO (from the sparse package).

And as far as I can tell, our wrapper class (which is the reason why you don't get the memory error on open) does not define nbytes, so at the moment there's no way to do that. You could try using dask, though, which does allow working with bigger-than-memory data.

there's two things that happen in as_shared_dtype (which may not be good design, and we should probably consider splitting it into as_shared_dtype and as_compatible_arrays or something): first, we cast everything to an array, then decide on a common dtype and cast everything to that.

The latter could easily be done by using numpy scalars, which as far as I can tell would be supported by most array libraries, including cupy. However, the reason we need to cast to arrays is that the array API (i.e. __array_namespace__) does not allow using scalars of any type, e.g. np.array_api.where (this is important for libraries that don't implement __array_ufunc__ / __array_function__). To clarify, what we're trying to support is something like python import numpy.array_api as np np.where(cond, cupy_array, python_scalar) which (intentionally?) does not work.

At the moment, as_shared_dtype (or, really, the hypothetical as_compatible_arrays) correctly casts python_scalar to a 0-d cupy.array for the example above, but if we were to replace cupy_array with chunked_cupy_array or chunked_cupy_array_with_units, the special casing for cupy stops to work and scalars will be cast to 0-d numpy.array. Conceptually, I tend to think of 0-d arrays as equivalent to scalars, hence the suggestion to have cupy treat numpy scalars and 0-d numpy.array the same way (I don't follow the array api closely enough to know whether that was already discussed and rejected).

So really, my question is: how do we support python scalars for libraries that only implement __array_namespace__, given that stopping to do so would be a major breaking change?

Of course, I would prefer removing the special casing for specific libraries, but I wouldn't be opposed to keeping the existing one. I guess as a short-term fix we could just pull _meta out of duck dask arrays and determine the common array type for that (the downside is that we'd add another special case for dask, which in another PR we're actually trying to remove).

As a long-term fix I guess we'd need to revive the stalled nested duck array discussion.

According to #503, there's two reasons: the fact that for masking this is so much easier to write (including when method chaining), and it is also API inherited from pandas.

to elaborate on that, xr.DataArray.where uses self as "then" and the second argument as "other", while for xr.where the second argument is "then" and the third argument is "other" (the condition is the first argument for both).

This API allows masking using just python arr.where(mask)

Is the policy working as designed

yes, it is: we guarantee support for at least 24 months, and only drop support once there's another version of python that was released more than 24 months ago. For example, python 3.8 was initially released on Oct 14, 2019 and python 3.9 was released on Oct 5, 2020. According to our policy we were able to drop python 3.8 for releases after Oct 5, 2022, since that's when python 3.9 was released 24 months ago.

This works very well for infrequent releases, since it guarantees that we don't accidentally require a very new version immediately after its release. However, these admittedly a bit complicated rules make interpreting the policy a bit more challenging than a simple "X months from this release" would for projects with frequent releases. Maybe we should add a (automatically created) support table for the core dependencies to the installation guide to make reasoning about the policy easier?

Python: 24 months from the last bugfix release (security releases are not considered).

That would make the support window less predictable, since the python devs might consider an additional bugfix release depending on the situation (there's a reason why the release peps say, emphasis mine: "X will receive bugfix updates approximately every 2 months for approximately 18 months"). Instead, maybe we should extend the support for python versions by about 6 months, to a total of 30 months? That would effectively align us with NEP-29, which is our upper limit anyways since that's what our dependencies follow (even if their releases don't usually happen at exactly that date).

And before anyone claims we're dropping support for a python version just because our policy tells us to: I'm excited about a number of changes to python, like the dict union and removeprefix / removesuffix in 3.9, the union types in 3.10, and the exception groups in 3.11, so really there is a compelling reason to upgrade as soon as the policy allows for each release.

flaky segfaults aside, we're down to just the zarr v3 tests, a flaky cdms2 test, and a test related to pint (though that one appears to be an upstream issue – not entirely sure, though).

I'm on a train wifi, so not really better. However, I think this is because xarray=2023.04.0 is not on conda-forge, yet (the PR to the feedstock still has to be merged), so you can't install xarray and pandas=2 into the same environment. As a workaround, you can try installing xarray using pip.

can you share a bit more about the environment you're trying to create? Is that by chance a py38 environment, or does one of the libraries you're trying to install have a upper bound for xarray? xarray>=2023.04.0 does not have the pin on pandas anymore, so you should be able to install that even if the repodata patches didn't work.

The issue is that here: https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/core/duck_array_ops.py#L193-L206 we try to convert everything to the same dtype, casting numpy and python scalars to an array. The latter is important, because e.g. numpy.array_api.where only accepts arrays as input.

However, detecting cupy beneath (multiple) layers of duckarrays is not easy, which means that for example passing a pint(dask(cupy)) array together with scalars will currently cast the scalars to 0-d numpy arrays, while passing a cupy array instead will result in 0-d cupy arrays.

My naive suggestion was to treat np.int64(0) and np.array(0, dtype="int64") the same, where at the moment the latter would fail for the same reason as np.array([0], dtype="int64").

no, mypy should be fine, as the CI version does not complain and I assume whatever the hook is reporting is also present on main (I didn't check, though).

As far as I can tell, the only thing left is another round of reviews.

I think we could check if the newest mypy still segfaults or not...

it didn't for me when I ran the hook, but that might just have been luck. To confirm, we'd need a PR that unpins mypy and potentially fixes all the errors that popped up since.

The mypy hook didn't take that long, but I guess it's the accumulated time that counts. Also, I learned today that the black-jupyter hook does all that black does plus format notebooks, so removing id: black should also shave off a bit.

@Illviljan, @headtr1ck, I just noticed that the CI version of mypy is pinned to <0.990, but the pre-commit hook is at 1.1.1. Is that intentional / known? Running the hook also exposes quite a few typing errors.

Otherwise this should be ready for a final review, and the failing datetime-related tests will be fixed by #7731.

sorry, I forgot about being able to use the upstream-dev environment for testing. That should solve just fine since by design it ignores all pinned dependencies, but we currently get occasional segfaults and unrelated failing tests, and we only test on a single OS / python version.

to properly test this, I guess we'd need to merge #7724 first?

It seems this fixes the failing tests unrelated to the datetime64 range.

Regarding the failing doctests, I decided to cast the values of arr.dt.<field> to int64 to keep the API unchanged. However, if we decide to follow pandas and return int32 I'm happy to make that change.

I think we'd need to be able to remove this xfail first: https://github.com/pydata/xarray/blob/f8127fc9ad24fe8b41cce9f891ab2c98eb2c679a/xarray/tests/test_backends.py#L712-L714 I didn't check, this could be xpassing right now.

CI says these are the tests we'd need to fix: FAILED xarray/tests/test_coding_times.py::test_should_cftime_be_used_source_outside_range - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[360_day] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[365_day] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[366_day] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[all_leap] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[gregorian] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[julian] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[noleap] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[proleptic_gregorian] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_cftimeindex.py::test_to_datetimeindex_out_of_range[standard] - Failed: DID NOT RAISE <class 'ValueError'> FAILED xarray/tests/test_dataarray.py::TestDataArray::test_sel_float - NotImplementedError: float16 indexes are not supported

Edit: one more on windows: FAILED xarray/tests/test_utils.py::test_maybe_coerce_to_str[1-2-expected1] - AssertionError: assert dtype('int64') == dtype('int32') + where dtype('int64') = Index([1, 2], dtype='int64').dtype + and dtype('int32') = Index([1, 2], dtype='int32').dtype Edit2: the doctests also fail: FAILED xarray/core/accessor_dt.py::xarray.core.accessor_dt.DatetimeAccessor FAILED xarray/core/accessor_dt.py::xarray.core.accessor_dt.TimedeltaAccessor FAILED xarray/core/dataarray.py::xarray.core.dataarray.DataArray.groupby

For test_should_cftime_be_used_source_outside_range and test_to_datetimeindex_out_of_range I'd probably use a date that is outside the s resolution range (not sure if that actually makes sense, though). What do you think, @spencerkclark?

For test_maybe_coerce_to_str I think the reason is that we use np.array to cast a python int to an array, but the default resolution is different on windows. Apparently, pandas still uses int64 if constructed directly from python ints, but numpy uses int32 on windows, and as you say pandas does not insist on int64 anymore. The fix would be to explicitly specify the dtype in the array calls.

And finally, I'm not sure what to do with test_sel_float. Maybe we can split the monolithic test into one parametrized by dtype and skip the float16 test variant for pandas>=2.0?

with the merge of #7441 we should already support pandas=2.0 on main. I think we can try unpinning in a PR to see how many of the errors from #7707 are related to pandas (none, hopefully, but I'm not sure).

it seems the tests segfaulted again. Not sure which test exactly is causing that, though.

let's close this and have the CI open a new issue for the other failing tests. I will point out, though, that the most recent run appears to have segfaulted.

the commit with my fix seems to have passed all CI (although using isoformat(sep=" ") is much better), and the upstream-dev CI only failed with unrelated issues, so once CI passes I think we can finally merge this.

the new version is on conda-forge, rerunning the failed CI should work now.

there's a bugfix release of sphinx-book-theme that has been released about two hours ago, we're just waiting on that to propagate to conda-forge.

As far as I understand #5954 and the meeting notes, we didn't decide anything about that yet (I might be wrong, though). So I assume the only reason we don't have write backends and save_dataset / save_dataarray right now is that nobody got around to doing that yet.

I guess the alternative would be to go with xr.save_dataset / xr.save_dataarray, but that seems like a lot of work (and I agree that since we already have Dataset.to_zarr we can also have DataArray.to_zarr)

see also #6323 and #4817

that looks like a good start, thanks.

I wonder if we should change the structure to start with an overview, then link to in-depth guides (just throwing out ideas): 1. project structure - code - docs - packaging (?) - automation 2. high-level description of a typical workflow (just list the steps and link to relevant documentation): - setup environment - create branch - edit files - run tests / build docs / run doctests (plus point to CI: for small changes no need to build manually) - commit - push to fork - create PR - what to do for very short changes (e.g. typo fixes) 3. set up environments (purpose: code including doctests, docs) - install dependencies (including special things to do for each OS) - install xarray 4. code standards and conventions 5. tests 6. typing 7. docstring conventions - format: numpydoc - examples (doctests) - see also 9. how to run tests 10. how to build docs 11. how to run doctests 12. special stuff when committing - pre-commit (and the pre-commit-ci bot) - commit message tags for controlling CI 13. PR title and message (short section on what to put into the PR description / title)

If that would be easier to follow, we could also split 2) into multiple parts: workflow for code changes (including doctest), workflow for documentation changes. And since we most likely will never be the best source for documentation on general git / github, try to link to other sources like https://docs.github.com or https://git-scm.com

Edit: I guess we could also combine 5 and 9, and 7, 10 and 11

depends on the test, I guess. Most of them are related to one of the netcdf backends (not sure which, the tests don't specify that), test_dataarray_compute just checks _in_memory (no I/O involved), and the pydap tests use netcdf underneath. So I'd say the issue is with one of the netcdf backends (netcdf4, as that's first in the priority list).

I've also seen a drastic reduction in performance with HDF5 1.12.2 (both netcdf4 and h5netcdf) on one of my colleague's datasets, so maybe that's much more visible on a mac? That doesn't explain the slow test_dataarray_compute, though.

Do we isolate the dask scheduler in any way? I assume that makes use of the builtin (non-distributed) scheduler?

Is it for a particular backend?

Not sure I understand. Can you elaborate, please?

since the macos 3.11 CI is not required, I'm tempted to merge this now, and continue to debug elsewhere.

should we use a released version of actions/labeler? A change to their main branch seems to have broken our workflow, but that's something to be expected, I guess.

Edit: turns out we had a workaround in our workflow, which has been fixed upstream (but broke the workaround). I think we have two options: keep the workaround but use v4, or stay on main and remove the workaround. The latter seems better in the long term, but staying on main involves the risk of unexpected changes.

I guess that means that the CPUs of the windows and macos runners are just slow, or there's other tasks that get prioritized, or something. All of this results in the tests being pretty flaky, so I'm not sure what we can do about it. I don't have access to any, but maybe someone with a mac could try to reproduce?

In any case, I'll increase the timeout a bit again since I think a timeout after ~1hour is better than the CI job being cancelled after 6 hours.

we don't usually assign issues, so you're free to just submit a pull request (don't forget to add tests that verify the new encoding actually works, though)

apparently python 3.11 on windows is also pretty slow, which makes the timeouts appear there as well. But in any case, here's the affected tests: FAILED xarray/tests/test_backends.py::TestDask::test_dask_roundtrip - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestDask::test_deterministic_names - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestDask::test_dataarray_compute - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestDask::test_load_dataset - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestDask::test_load_dataarray - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestDask::test_inline_array - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestPydap::test_cmp_local_file - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestPydap::test_compatible_to_netcdf - Failed: Timeout >90.0s FAILED xarray/tests/test_backends.py::TestPydap::test_dask - Failed: Timeout >90.0s which are the exact same tests that also fail for >300s, so I'm guessing those are the ones that stall.

Does anyone know why those take so long only on macos (and, partially, windows)? Does that have to do with the runner hardware?

you can also pass a dict, so nc.loc[{"time": dt}] or nc.loc[dict(time=dt)] should work as well

this looks good to me, but we'd need some tests to make sure we don't reintroduce this issue. I'd probably extend the "simple" case of xarray/tests/test_computation.py::test_polyval by creating two versions: one with int32 values for degree and one with int64 values

this happens with any code that calls polyval on coefficients with int32 dtypes. The example from the docs: ```python In [19]: import xarray as xr ...: ...: x = xr.DataArray(np.arange(10), dims=["x"], name="x") ...: a = xr.DataArray(3 + 4 * x, dims=["x"], coords={"x": x}) ...: out = a.polyfit(dim="x", deg=1, full=True) ...: out Out[19]: <xarray.Dataset> Dimensions: (degree: 2) Coordinates: * degree (degree) int64 1 0 Data variables: x_matrix_rank int64 2 x_singular_values (degree) float64 1.358 0.3963 polyfit_coefficients (degree) float64 4.0 3.0 polyfit_residuals float64 4.522e-28

In [20]: xr.polyval(coord=x, coeffs=out.assign_coords(degree=out.degree.astype("int64"))) Out[20]: <xarray.Dataset> Dimensions: (x: 10) Dimensions without coordinates: x Data variables: x_matrix_rank (x) int64 2 4 6 8 10 12 14 16 18 20 x_singular_values (x) float64 0.3963 1.754 3.111 ... 9.899 11.26 12.61 polyfit_coefficients (x) float64 3.0 7.0 11.0 15.0 ... 27.0 31.0 35.0 39.0 polyfit_residuals (x) float64 4.522e-28 9.044e-28 ... 4.07e-27 4.522e-27

In [21]: xr.polyval(coord=x, coeffs=out.assign_coords(degree=out.degree.astype("int32")))

ValueError Traceback (most recent call last) Cell In[21], line 1 ----> 1 xr.polyval(coord=x, coeffs=out.assign_coords(degree=out.degree.astype("int32")))

File .../xarray/core/computation.py:1972, in polyval(coord, coeffs, degree_dim) 1968 raise ValueError( 1969 f"Dimension {degree_dim} should be a coordinate variable with labels." 1970 ) 1971 if not np.issubdtype(coeffs[degree_dim].dtype, int): -> 1972 raise ValueError( 1973 f"Dimension {degree_dim} should be of integer dtype. Received {coeffs[degree_dim].dtype} instead." 1974 ) 1975 max_deg = coeffs[degree_dim].max().item() 1976 coeffs = coeffs.reindex( 1977 {degree_dim: np.arange(max_deg + 1)}, fill_value=0, copy=False 1978 )

ValueError: Dimension degree should be of integer dtype. Received int32 instead. ```

thanks for the fix and welcome to xarray, @Ravenin7

as discussed on discord, the reason for that failure is that current versions of numba are incompatible with numpy>=1.24. Downgrading to numpy<1.24 should get the docs to build successfully.

closed by #7444

Does it still exist?

It does not (I think?): the wiki feature is disabled, and enabling it just creates a blank instance. So I guess if it ever existed it is gone now.

I agree, those do indeed seem unrelated, and I have no idea why the python 3.11 macos CI takes that long to run.

pre-commit.ci run

the docs failure is real, I think we need to update https://github.com/pydata/xarray/blob/821dc24b5f3ed91b843a634bf8513a26046269ef/doc/user-guide/weather-climate.rst#L236 to use the new parameter (not sure how exactly, though)

Since the label exists and the spelling is correct, I think the only thing we need to do here is use the appropriate sphinx role. To figure out which one, we can either manually search the intersphinx registry using bash $ python -m sphinx.ext.intersphinx https://docs.xarray.dev/en/latest/objects.inv | less (which is a bit tedious because that particular section is pretty big, and we need to have sphinx installed)

or we can use sphobjinv: ```bash $ sphobjinv suggest -u https://docs.xarray.dev/en/latest/ datetime_component_indexing

Attempting https://docs.xarray.dev/en/latest ... ... no recognized inventory. Attempting "https://docs.xarray.dev/en/latest/objects.inv" ... ... inventory found.

The intersphinx_mapping for this docset is LIKELY:

(https://docs.xarray.dev/en/latest/, None)

Project: xarray Version: 2023.2.1.dev19

7468 objects in inventory.

1 result found at/above current threshold of 75.

:std:label:datetime_component_indexing `` which helpfully tells us the correct link (though I think we can use:label:instead of:std:label:`)

there's also a few failing zarr v3 tests (cc @jhamman): xarray/tests/test_backends.py::TestZarrDirectoryStoreV3::test_write_read_select_write: KeyError: 'var1' xarray/tests/test_backends.py::TestZarrDirectoryStoreV3FromPath::test_write_read_select_write: KeyError: 'var2'

and some unrelated tests that didn't raise errors when we expected them (not sure what causes that, I didn't investigate).

the currently failing pint tests are an upstream issues (numpy dev has changed amin and amax to become aliases of min and max instead of the other way around, and round_ has been deprecated in favor of round) and will be fixed by hgrecco/pint#1721

both the failing upstream-dev CI and the failing docs CI seem to be unrelated to this. The TimeResampleGrouper seems to do something different from the one in #7561, but I think that's up to that PR to figure out (since we don't yet expose it as public API).

So considering that, how much do you think is needed until we can merge this? And can I help with anything?

thanks, that helps. However, it does not confirm my suspicion since all data variables are already in float64, and thus they shouldn't change dtypes. Could you also post the repr (either the text or the html repr should be sufficient) of a, and maybe also the file type of the file you're loading the dataset from (ifileFamBulk)?

I can't really tell from the information you posted so far. Could you post the repr of da_fam_bulk (print(da_fam_bulk) or display(da_fam_bulk) using ipython / jupyter, plus maybe a screenshot of the HTML repr if you're in a notebook)?

I do suspect, however, that da_fam_bulk has a dtype that is not float64, but where will use float64(nan) as a fill value, casting the entire array to a dtype that has a much higher memory usage.

I just encountered this one as well. It seems the issue is here: https://github.com/pydata/xarray/blob/43ba095712de12c957e0a4acf956df01d84b2046/xarray/core/variable.py#L1815-L1820 where fill_value = np.nan and dtype = "int64".

As mentioned in https://github.com/numpy/numpy/issues/8017#issuecomment-1155517077, numpy now warns instead of silently casting nan to int.

Can we do anything about that here (besides silencing the warning, but I'm not sure if that actually makes sense), or do we need to lobby for nullable dtypes in numpy or a numpy-adjacent library?

Edit: as it seems there are at least 4 deferred NEPs (NEPs 12, 24, 25, and 26) on the topic of missing values this might be more tricky than I expected. So I guess that means that we might have to try finding a workaround.

with the release of sphinx-book-theme=1.0.0 yesterday we can finally move this forward.

In the process I fixed a few other docs issues so I'll merge this now to avoid having the docs CI fail on main.

don't worry about the commit history, we always squash merge

here's a example of how we solved this before: https://github.com/pydata/xarray/blob/4427eaaf418103eca8786af4d97f3d365cbcba63/xarray/core/dataset.py#L2093-L2094

This is technically the same, except when reading this you don't have to parse the rst link syntax.

Apologies for letting this sit for so long.

The reason for the unexpected behavior seems to be that mean is implemented using sum / count: https://github.com/pydata/xarray/blob/21d86450b3cec595c74aa410cbcc367c9c7f8a0a/xarray/core/rolling.py#L176 https://github.com/pydata/xarray/blob/21d86450b3cec595c74aa410cbcc367c9c7f8a0a/xarray/core/rolling.py#L161-L163 where min_periods is applied in the sum (by masking values where count < min_periods).

However, sum on rolling objects will fill any missing values with 0 before doing anything else, so when the actual sum is computed skipna does not have any effect.

So if you were to set min_periods=1 you'd get the same result as what you'd expect while min_periods=3 is what you're seeing.

@pydata/xarray, any idea what to do here? Should we document that passing skipna does not have any effect on rolling window sums / means, or would it be better to change the implementation? Or maybe I'm missing something else?

Could you post the output you get? If I try it locally, I get: python In [3]: import xarray as xr ...: import pandas as pd ...: import numpy as np ...: ...: da = xr.DataArray( ...: np.array([1, 2, 3, 1, 2, np.nan]), ...: dims="time", ...: coords=dict( ...: time=("time", pd.date_range("01-01-2001", freq="M", periods=6)), ...: labels=("time", np.array(["a", "b", "c", "c", "b", "a"])), ...: ), ...: ) ...: da.labels.attrs["units"] = "a.u." ...: ...: ds = xr.Dataset(dict(da=da)) ...: ds.attrs["units"] = "V" ...: display(ds) ...: ...: m = ds.groupby("labels").mean(keep_attrs=True) ...: display(m) ...: display(m.labels) # labels has attribute for units <xarray.Dataset> Dimensions: (time: 6) Coordinates: * time (time) datetime64[ns] 2001-01-31 2001-02-28 ... 2001-06-30 labels (time) <U1 'a' 'b' 'c' 'c' 'b' 'a' Data variables: da (time) float64 1.0 2.0 3.0 1.0 2.0 nan Attributes: units: V <xarray.Dataset> Dimensions: (labels: 3) Coordinates: * labels (labels) object 'a' 'b' 'c' Data variables: da (labels) float64 1.0 2.0 2.0 Attributes: units: V <xarray.DataArray 'labels' (labels: 3)> array(['a', 'b', 'c'], dtype=object) Coordinates: * labels (labels) object 'a' 'b' 'c' Attributes: units: a.u. which looks fine to me? But maybe I'm missing something?

It seems that skipna has been ignored since #2541 (more than four years ago). While that might have been unintentional at the time I don't think resample should have that option, so maybe we should just raise a warning like we do for keep_attrs and eventually remove both.

Does this do what you expect? ```python In [2]: da.resample(time="A-Dec").mean(skipna=True) Out[2]: <xarray.DataArray (time: 1, value: 1)> array([[6.5]]) Coordinates: * value (value) int64 0 * time (time) datetime64[ns] 2022-12-31

In [3]: da.resample(time="A-Dec").mean(skipna=False) Out[3]: <xarray.DataArray (time: 1, value: 1)> array([[nan]]) Coordinates: * value (value) int64 0 * time (time) datetime64[ns] 2022-12-31 ```

Edit: I don't think this is related to #6772, but I'll also have a look at what's wrong there

hmm... would np.nan work?

not sure about alignment, but at least obj.sizes would break with multiple dims: while it does not compare as equal, the hash() of np.nan stays constant (same with any other object, I guess).

I'd probably use itertools.count() or the uuid module to generate globally unique dimension names... something like ```python class UnnamedDimensions: def __init__(self): self.dimension_names = (f"unnamed_dim{number}" for number in itertools.count())

def __call__(self, n):
    return list(itertools.islice(dimension_names, None, n))

unnamed_dimensions = _UnnamedDimensions() or using `uuid` (probably overkill):python def unnamed_dimensions(n): return [uuid.uuid4() for _ in range(n)] you'd use it like this:python d1 = xr.DataArray(data=[1, 2], dims=unnamed_dimensions(1)) d2 = xr.DataArray(data=[[1, 2]], dims=unnamed_dimensions(2)) ``` which would make "unnamed" a bit more explicit.

Edit: that's probably not so different from what you meant with d1_i

as expected that environment is not solvable because sphinx-book-theme didn't have a release since the bump of the sphinx version

I think we should pin sphinx-book-theme to some appropriate version, right now we have 0.0.41, which is about two years old (and is probably the last version that does not contain the upper bound on sphinx)

thanks, that makes sense to me. I think that rules out option 3, so now we only need to decide whether we remove only the data url or the entire section (personally, I would lean towards the latter).

1 is the status quo (minus the removal of the data server url): the cells are set to "raw", so they are not run. We don't want to hit the server during CI anyways, as hitting the server would slow down the docs build too much.

It seems like there's at least two people who tried to follow the subsetting (or otherwise play with the raw data), so the section does have some value. However, I don't know how this compares to the burden of maintaining it, especially since we don't run it in CI.

we are casting everything back to datetime64[ns] when creating xarray objects, for example, so the only way to even get a non-nanosecond datetime variable is (or was, we might have fixed that?) through the zarr backend (though that would / might fail elsewhere).

@spencerkclark knows much more about this, but in any case we're aware of the change and are working it (see e.g. #7441). (To be fair, though, at the moment it is mostly Spencer who's working on it, and he seems to be pretty preoccupied.)

don't we do that already? As far as I can tell, the url is there to explain where we got the data from and what preprocessing was done. Unless I'm missing something?

the data server is out of our control, so we can't fix this ourselves, unfortunately.

We might be able to look for a different server for that model, but I don't know enough about the model to do that myself. Any ideas, @pydata/xarray?

this looks like a duplicate of #7486. Closing, but feel free to follow the discussion there.

iris has the pin in their package metadata

should this have been closed by #7283?