Thank you @keewis

thank you for your patience, @igibek! i've enabled the private vulnerability reporting for the repo.

@maxaragon, i'm curious. what version of xarray/netcdf4 are you using? i'm asking because this appears to be working fine on my end

```python In [1]: import xarray as xr

In [2]: ds = xr.open_dataset("20200825_hyytiala_icon-iglo-12-23.nc")

In [3]: ds Out[3]: <xarray.Dataset> Dimensions: (time: 25, level: 90, flux_level: 91, frequency: 2, soil_level: 9) Coordinates: * time (time) datetime64[ns] 2020-08-25 ... 2020-0... * level (level) float32 90.0 89.0 88.0 ... 3.0 2.0 1.0 * flux_level (flux_level) float32 91.0 90.0 ... 2.0 1.0 * frequency (frequency) float32 34.96 94.0 Dimensions without coordinates: soil_level Data variables: (12/62) latitude float32 ... longitude float32 ... altitude float32 ... horizontal_resolution float32 ... forecast_time (time) timedelta64[ns] ... height (time, level) float32 ... ... ... gas_atten (frequency, time, level) float32 ... specific_gas_atten (frequency, time, level) float32 ... specific_saturated_gas_atten (frequency, time, level) float32 ... specific_dry_gas_atten (frequency, time, level) float32 ... K2 (frequency, time, level) float32 ... specific_liquid_atten (frequency, time, level) float32 ... Attributes: (12/13) institution: Max Planck Institute for Meteorology/Deutscher Wette... references: see MPIM/DWD publications source: svn://xceh.dwd.de/for0adm/SVN_icon/tags/icon-2.6.0-n... Conventions: CF-1.7 location: hyytiala file_uuid: ace15f8ba477497c8d1dd0833b5ac674 ... ... year: 2020 month: 08 day: 25 history: 2021-01-25 08:24:29 - File content harmonized by the... title: Model file from Hyytiala pid: https://hdl.handle.net/21.12132/1.ace15f8ba477497c ```

here are the versions i'm using

```python In [4]: xr.show_versions() /Users/andersy005/mambaforge/envs/playground/lib/python3.10/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit: None python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:41:22) [Clang 13.0.1 ] python-bits: 64 OS: Darwin OS-release: 22.1.0 machine: arm64 processor: arm byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1

xarray: 2022.10.0 pandas: 1.5.1 numpy: 1.23.4 scipy: 1.9.3 netCDF4: 1.6.1 pydap: installed h5netcdf: 1.0.2 h5py: 3.7.0 Nio: None zarr: 2.13.3 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2022.10.2 distributed: 2022.10.2 matplotlib: 3.6.1 cartopy: None seaborn: 0.12.0 numbagg: None fsspec: 2022.10.0 cupy: None pint: 0.20.1 sparse: None flox: None numpy_groupies: None setuptools: 65.5.0 pip: 22.3 conda: None pytest: None IPython: 8.6.0 sphinx: None ```

@dependabot rebase

@dependabot squash and merge

@dependabot rebase

thank you for the clarification, @mwtoews

Remove check-manifest, since it isn't used

Could we keep check-manifest and add a proper check instead of removing it entirely? It's worth ensuring we are shipping everything and check-manifest would help catch missing files, etc...

@dependabot rebase

Duplicate of: - https://github.com/pydata/xarray/issues/6818

which seems to have been addressed (if you use the main branch).

Great... keep us posted once you have a working solution.

I'm going to convert this issue in a discussion instead.

@lassiterdc, writing large, chunked xarray dataset to a netCDF file is always a challenge and quite slow since the write is serial. However, you could take advantage of the xr.save_mfdataset() function to write to multiple netCDF files. here's a good example that showcase how to achieve this: https://ncar.github.io/esds/posts/2020/writing-multiple-netcdf-files-in-parallel-with-xarray-and-dask

Thank you for this fix, @jrbourbeau!

I opened an issue just in case this is something that can be fixed upstream.

• https://github.com/numba/numba.github.com/issues/10

It's not immediately clear to me why the docs build is failing. Has that been happening elsewhere?

When building docs on RTD, we turn Sphinx warnings into errors. Sphinx is throwing a warning that is then turned into an error after docs are built, as it appears that the SSL certificate for numba.pydata.org has expired or is misconfigured.

bash loading intersphinx inventory from https://rasterio.readthedocs.io/en/latest/objects.inv... loading intersphinx inventory from https://sparse.pydata.org/en/latest/objects.inv... WARNING: failed to reach any of the inventories with the following issues: intersphinx inventory 'https://numba.pydata.org/numba-doc/latest/objects.inv' not fetchable due to <class 'requests.exceptions.SSLError'>: HTTPSConnectionPool(host='numba.pydata.org', port=443): Max retries exceeded with url: /numba-doc/latest/objects.inv (Caused by SSLError(CertificateError("hostname 'numba.pydata.org' doesn't match either of '*.github.com', 'www.github.com', 'github.io', 'github.com', '*.github.io', 'githubusercontent.com', '*.githubusercontent.com'")))

@dependabot squash and merge

at SciPy i learned of fugue which tries to provide a unified API for distributed DataFrames on top of Spark and Dask. it could be a great source of inspiration.

@lassiterdc, in your example above, f_in_ncs is just a string and you are passing this string to xr.open_mfdataset() which i don't think knows what to do with it.

python f_in_ncs = "data/"

have you tried retrieving the list of all files under "data/" via the glob module?

```python import glob f_in_ncs = sorted(glob.glob("data/*.nc"))

mf_ds = xr.open_mfdataset(f_in_ncs, concat_dim = "time", chunks={'outlat':3500, 'outlon':7000, 'time':50}, combine = "nested", engine = 'netcdf4') ```

@dcherian, are you still working on this or can we merge it?

Ok, another option would be to add that to xncml

@andersy005 What do you think ?

@huard, I haven't touched the codebase in that repo for three years 😃... So, I'm happy to transfer the xncml repo to xarray-contrib org and give you and anyone who wants access to it

Thank you, @dcherian!

Thank you, @headtr1ck!

my bad :) thank you for reporting this in #6720. I'm going to look into what;s going on

Thank you, @delgadom!

The failures appear to be caused by recent changes introduced in

• https://github.com/pandas-dev/pandas/pull/47338

python ImportError while importing test module '/home/runner/work/xarray/xarray/xarray/tests/test_backends.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../../micromamba/envs/xarray-tests/lib/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) xarray/tests/test_backends.py:86: in <module> from .test_dataset import ( xarray/tests/test_dataset.py:14: in <module> from pandas.core.computation.ops import UndefinedVariableError

@mjwillson, have you looked at https://github.com/carbonplan/xarray-schema? xarray-schema provides some of the functionality you are looking for...

• Related to https://github.com/pydata/xarray/issues/1900

@PovedaGerman, this doesn't appear to be an Xarray issue. Do you mind asking on the xinvert issue tracker: https://github.com/miniufo/xinvert

Maybe we can merge those two in a future PR?

👍🏽

The CI failures appear to be related to https://github.com/dask/dask/issues/9137

```bash file /home/runner/work/xarray/xarray/xarray/tests/test_distributed.py, line 82 @pytest.mark.parametrize("engine,nc_format", ENGINES_AND_FORMATS) def test_dask_distributed_netcdf_roundtrip( file /usr/share/miniconda/envs/xarray-tests/lib/python3.10/site-packages/distributed/utils_test.py, line 138 @pytest.fixture def loop(cleanup): E fixture 'cleanup' not found

  available fixtures: add_standard_imports, backend, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, loop, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, testrun_uid, tmp_netcdf_filename, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
  use 'pytest --fixtures [testpath]' for help on them.

```

Ccing @jrbourbeau for visibility upstream...

It appears to have been fixed in

• https://github.com/pydata/xarray/pull/6581

Closing this. Feel free to reopen if I missed something....

5734 has greatly improved the performance. Fantastic work @dcherian 👏🏽

```python In [13]: import xarray as xr, pandas as pd, numpy as np

In [14]: ds = xr.Dataset({"a": xr.DataArray(np.r_[np.arange(500.), np.arange(500.)]), ...: "b": xr.DataArray(np.arange(1000.))})

In [15]: ds Out[15]: <xarray.Dataset> Dimensions: (dim_0: 1000) Dimensions without coordinates: dim_0 Data variables: a (dim_0) float64 0.0 1.0 2.0 3.0 4.0 ... 496.0 497.0 498.0 499.0 b (dim_0) float64 0.0 1.0 2.0 3.0 4.0 ... 996.0 997.0 998.0 999.0 ```

```python In [16]: xr.set_options(use_flox=True) Out[16]: <xarray.core.options.set_options at 0x104de21a0>

In [17]: %%timeit ...: ds.groupby("a").mean() ...: ...: 1.5 ms ± 3.3 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [18]: xr.set_options(use_flox=False) Out[18]: <xarray.core.options.set_options at 0x144382350>

In [19]: %%timeit ...: ds.groupby("a").mean() ...: ...: 94 ms ± 715 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) ```

• https://github.com/pydata/xarray/pull/5560 introduced "use_bottleneck" option to disable/enable using bottleneck. can we close this issue or keep it open?

ImportError: Pandas requires version '0.15.1' or newer of 'xarray' (version '0.1.dev1+gfdf7303' currently installed).

looking at the reported xarray version, i'm curious... 🧐 are you using a shallow git clone of xarray? I'm able to reproduce the version issue via these steps:

bash git clone --depth 1 git@github.com:pydata/xarray.git cd xarray python -m pip install -e .

```bash conda list xarray ─╯

packages in environment at /Users/andersy005/mambaforge/envs/test:

Name Version Build Channel

xarray 0.1.dev1+g126051f dev_0 <develop> ```

this appears to have been fixed in the latest alpha release of rasterio v1.3a3 (which isn't available on conda-forge but is available on PyPI) via fsspec

• https://github.com/rasterio/rasterio/issues/2360
• https://github.com/corteva/rioxarray/issues/440

```python In [13]: import xarray as xr

In [14]: import fsspec

In [15]: import rasterio

In [16]: rasterio.version Out[16]: '1.3a3'

In [17]: with fsspec.open('2d_test.tiff') as f: ...: ds = xr.open_dataset(f, engine='rasterio') ...:

In [18]: ds Out[18]: <xarray.Dataset> Dimensions: (band: 1, y: 10, x: 10) Coordinates: * band (band) int64 1 xc (y, x) float64 ... yc (y, x) float64 ... spatial_ref int64 ... Dimensions without coordinates: y, x Data variables: band_data (band, y, x) float64 ... ```

@dependabot squash and merge

@dependabot squash and merge

Thank you, @stanwest!

@ArcticSnow,

The value z is a float32 which varies from 2000 to -2000 along the time dimension. After being saved in the subsample, z is still a float32 but the values that are less than -1000 are being offset by 44500.

You may have scale_factor and add_offset attributes in your dataset.

However, if I do (ds.z.isel(latitude[1,2,3], longitude=[3,4,5])*1).to_netcdf('sub.nc')

There's a chance xarray is discarding the attributes/encoding during the ds.z.isel(latitude[1,2,3], longitude=[3,4,5])*1 and as a result, netCDF ends up not encoding z during the to_netcdf() call.

What's the output of

python print(ds.z.encoding) print(ds.z.attrs) ??

We'll also need to keep an eye on the asv benchmarks to ensure that there's no major regression in performance (I haven't checked yet).

I enabled the asv-benchmarks workflow via the run-benchmark label. Feel free to turn this off (by removing the label) if the PR isn't ready :)

Closed by #6224

after twine check where we pip install and then try to import xarray. Alternatively we could have another test config in our regular CI to build + import.

it's my understanding that we're doing this here:

https://github.com/pydata/xarray/blob/b09de8195a9e22dd35d1b7ed608ea15dad0806ef/.github/workflows/pypi-release.yaml#L74-L80

However, it appears that the three seconds in sleep 3 aren't enough to warrant getting the latest metadata from test.pypi.org... For instance, the latest build ended up pulling v0.20.2 instead of the v0.21.0 (relevant run)

@haritha1022, thank you for the report!

i am unable to remove the color bar. i want to remove the color bar

The issue lies in this line:

python z.plot.pcolormesh(cmap="turbo",vmin=7500,vmax=8500, ax = ax1,cbar=False)

The right argument name is add_colorbar instead of cbar:

Under "Creating a Data Array" and next to the subitem "coords", it is written, that you can use a dictionary like {'coo1': [1,2,3], 'coo2': ['a', 'b', 'c'], ...}, so you can later skip the "dims"

Are you referring to this page? Per this documentation page, dims are only inferred from coords when coords is a list of tuples.

dims: a list of dimension names. If omitted and coords is a list of tuples, dimension names are taken from coords.

So, I'd say that the ValueError you get when using coords defined in a dictionary without specifying dims isn't really a bug.

But when I tried this, I get the message, that using a dictionary for coords is outdated.

What do you think it should return?

@zxdawn,

Thank you for the thorough, reproducible example.

Per Scipy's documentation, RegularGridInterpolator uses fill_value=None for extrapolation

```python In [23]: da.interp(x=x, y=y, kwargs={"fill_value": None})

Out[23]: <xarray.DataArray (z: 3)> array([-0.26074336, 0.63496063, -0.46643289]) Coordinates: x (z) float64 -0.5 1.5 2.5 y (z) float64 0.15 0.25 0.35 Dimensions without coordinates: z ```

Do others have thoughts here? I would support stripping the leading zeros from the MM part of the version string in favor of consistency here.

I'm in favor of a non-zero-padded version for the benefit of having canonical/normalized versions that also match git tags/history

Okay, I believe this is ready. I may have missed some corner cases (I'm happy to address these in separate PRs).

If I'm not mistaken, this appears to have been addressed by https://github.com/pydata/xarray/pull/2659. Should we close this issue?

• xref the more recent issue: https://github.com/pandera-dev/pandera/issues/705 which aims to implement a pandera.xarray module within pandera

@theobarnhart-USGS, did you find a solution and/or can we close this issue?

@shoyer Tried Dask as you suggested, and it helped significantly! Thanks for the suggestion!

@zachglee, should we close this issue?

Ha, thanks. It makes sense now. Shall we close this?

Great! I'm closing this for the time being...

@mroeschke, could you add an entry to the https://github.com/pydata/xarray/blob/main/doc/whats-new.rst?

Okay... I think the following comment is still valid:

The issue appears to be caused by the coordinates which are used in dask_tokenize

It appears that the deterministic behavior of the tokenization process is affected depending on whether the dataset/datarray contains non-dimension coordinates or dimension coordinates

python In [2]: ds = xr.tutorial.open_dataset('rasm')

```python In [39]: a = ds.isel(time=0)

In [40]: a Out[40]: <xarray.Dataset> Dimensions: (y: 205, x: 275) Coordinates: time object 1980-09-16 12:00:00 xc (y, x) float64 189.2 189.4 189.6 189.7 ... 17.65 17.4 17.15 16.91 yc (y, x) float64 16.53 16.78 17.02 17.27 ... 28.26 28.01 27.76 27.51 Dimensions without coordinates: y, x Data variables: Tair (y, x) float64 ...

In [41]: dask.base.tokenize(a) == dask.base.tokenize(a) Out[41]: True ```

```python In [42]: b = ds.isel(y=0)

In [43]: b Out[43]: <xarray.Dataset> Dimensions: (time: 36, x: 275) Coordinates: * time (time) object 1980-09-16 12:00:00 ... 1983-08-17 00:00:00 xc (x) float64 189.2 189.4 189.6 189.7 ... 293.5 293.8 294.0 294.3 yc (x) float64 16.53 16.78 17.02 17.27 ... 27.61 27.36 27.12 26.87 Dimensions without coordinates: x Data variables: Tair (time, x) float64 ...

In [44]: dask.base.tokenize(b) == dask.base.tokenize(b) Out[44]: False ```

This looks like a bug in my opinion...

The issue appears to be caused by the coordinates which are used in dask_tokenize

I tried running the reproducer above and things seem to be working fine. I can't for the life of me understand why I got non-deterministic behavior four hours ago :(

```python In [1]: import dask, xarray as xr

In [2]: ds = xr.tutorial.open_dataset('rasm')

In [3]: dask.base.tokenize(ds) == dask.base.tokenize(ds) Out[3]: True

In [4]: dask.base.tokenize(ds.Tair._coords) == dask.base.tokenize(ds.Tair._coords) Out[4]: True ```

```python In [5]: xr.show_versions()

INSTALLED VERSIONS

commit: None python: 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 20:33:18) [Clang 11.1.0 ] python-bits: 64 OS: Darwin OS-release: 20.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.1 libnetcdf: 4.8.1

xarray: 0.20.1 pandas: 1.3.4 numpy: 1.20.3 scipy: 1.7.3 netCDF4: 1.5.8 pydap: None h5netcdf: 0.11.0 h5py: 3.6.0 Nio: None zarr: 2.10.3 cftime: 1.5.1.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2021.11.2 distributed: 2021.11.2 matplotlib: 3.5.0 cartopy: None seaborn: None numbagg: None fsspec: 2021.11.1 cupy: None pint: 0.18 sparse: None setuptools: 59.4.0 pip: 21.3.1 conda: None pytest: None IPython: 7.30.0 sphinx: 4.3.1 ```

Note that I can't reproduce it using this example:

I could be wrong but it appears that when you introduce a _FillValue in your dataarray, you end up with the same outcome:

```python In [53]: import numpy as np ...: import xarray as xr ...: ...: da = xr.DataArray(np.array([1,2,4294967295], dtype='uint')).rename('test_array')

In [56]: da.encoding['_FillValue'] = 4294967295 ```

```python In [62]: da.to_netcdf("test.nc", engine='netcdf4')

In [63]: !ncdump -h test.nc netcdf test { dimensions: dim_0 = 3 ; variables: uint64 test_array(dim_0) ; test_array:_FillValue = 4294967295ULL ; data:

test_array = 1, 2, _ ; } ```

```python In [64]: d = Dataset("test.nc")

In [65]: d Out[65]: <class 'netCDF4._netCDF4.Dataset'> root group (NETCDF4 data model, file format HDF5): dimensions(sizes): dim_0(3) variables(dimensions): uint64 test_array(dim_0) groups:

In [66]: xr.open_dataset('test.nc') Out[66]: <xarray.Dataset> Dimensions: (dim_0: 3) Dimensions without coordinates: dim_0 Data variables: test_array (dim_0) float64 ... ```

python In [67]: xr.open_dataset('test.nc').test_array Out[67]: <xarray.DataArray 'test_array' (dim_0: 3)> array([ 1., 2., nan]) Dimensions without coordinates: dim_0 Notice that xarray is using np.NaN as a sentinel value for the missing / fill_values. Because np.NaN is a float, this forces the entire array of integers to become floating pointing numbers...

@andersy005 if you can rely on dask always being present, dask.base.tokenize(xarray_object) will do what you want.

@dcherian, I just realized that dask.base.tokenize deosn't return a deterministic token for xarray objects:

```python In [2]: import dask, xarray as xr

In [3]: ds = xr.tutorial.open_dataset('rasm')

In [4]: dask.base.tokenize(ds) == dask.base.tokenize(ds) Out[4]: False

In [5]: dask.base.tokenize(ds) == dask.base.tokenize(ds) Out[5]: False ```

The issue appears to be caused by the coordinates which are used in __dask_tokenize__

https://github.com/pydata/xarray/blob/dbc02d4e51fe404e8b61656f2089efadbf99de28/xarray/core/dataarray.py#L870-L873

python In [8]: dask.base.tokenize(ds.Tair.data) == dask.base.tokenize(ds.Tair.data) Out[8]: True

python In [16]: dask.base.tokenize(ds.Tair._coords) == dask.base.tokenize(ds.Tair._coords) Out[16]: False

Is this the expected behavior or am I missing something?

The CI failure is related. It has moved the pytest.importorskip("distributed") below a distributed import. See PyCQA/isort#1840.

Good catch. I missed it.

The CI failures seem to be unrelated. Are we okay with merging this?

In any case, I'd vote to remove the pre-commit autoupdate CI as that has some quirks that make it difficult to use for everyone other than me.

Once we're satisfied with pre-commit.ci, we should remove https://github.com/pydata/xarray/blob/main/.github/workflows/ci-pre-commit.yml workflow as well.

Isn't this a bug?

Yes. Part of this is related to #5897

We could potentially still automate

"add new section to the whats-new.rst", "update the stable branch", "update the active version of the docs" (maybe?), and "email various mailing lists".

with more frequent releases, will we still need to maintain the "stable" branch? It's my understanding that the "stable" branch is slightly different from the latest tag due to very few, trivial/doc changes that are added between releases. Without this additional branch, we would have one less thing to automate.

This behavior doesn't feel intuitive to me. It does happen on the stable version as well so it isn't related to this PR. Tested on windows 10 and Firefox.

I'll look into this... It appears to be a CSS issue

👍🏽 for frequent releases... With more frequent releases, should we switch to calendar versioning? I remember seeing this discussion in one of the issues but I don't recall what the conclusion was.

This applies to pandas datetime objects as well (#5898)

Thank you for working on this, @mlhenderson! Seems there have been attempts at fixing this issue before

• https://github.com/pydata/xarray/pull/4053
• https://github.com/executablebooks/sphinx-book-theme/issues/238#issuecomment-714784321 which suggests using the .xr-wrap { display: block !important }. Not sure why this fix was implemented in sphinx-book-theme and not in xarray directly...

Ccing @benbovy who wrote the HTML repr in case he has any feedback.

github-script action v5 includes breaking changes that will cause the CI to fail.

https://github.com/pydata/xarray/blob/54dba584dcef37ac0713ed94cd99aa2d5237c197/.github/workflows/upstream-dev-ci.yaml#L161

must be updated to github.rest.issues.create

https://github.com/pydata/xarray/blob/54dba584dcef37ac0713ed94cd99aa2d5237c197/.github/workflows/upstream-dev-ci.yaml#L169

and github.rest.issues.update

Do we not want to discuss this in the bi-weekly dev meetings perhaps?

👍🏽 I'm in favor of discussing this in the dev meetings... Being able to easily switch plotting backends would be a nice feature to have from a user's perspective.

Closing this because it has gone stale :(

Readthedocs build is failing due to timeout. It appears that our doc builds have been failing for the last few hours, and my hunch is that somehow we are exceeding the 15 minutes build time limit for some PRs

@rabernat, the gentlest of bumps on this :)... How much work (content) is left to bring this to completion? I'm asking because I'd be happy to help if there's still more work and/or follow-up PR needed.

could someone check whether binder still works?

It's broken, unfortunately :(

I will submit a fix later today unless someone beats me to it

Pushing another commit to the same branch should cancel previous runs and start new runs (for the new commit). So, the fact that all runs were cancelled makes me think that it was a glitch somewhere :)

I take this back... I just realized that #5467 is a different PR :). The previous runs cancellation is currently triggered for commits in the same branch/PR

@max-sixty, I skimmed through Github Action logs, but I couldn't figure out where the "cancel" command came from. I am going to speculate that it was some glitch in GitHub action.

Is it something to do with pushing another commit to #5467?

Pushing another commit to the same branch should cancel previous runs and start new runs (for the new commit). So, the fact that all runs were cancelled makes me think that it was a glitch somewhere :)

Do we fire twice for each release? Maybe that's fine though?

Yes, on both push and release events. The publication to PyPI step is only run for the release event though i.e. if one were to push a tag to GitHub, this tag would never be published to PyPI via the GitHub workflow.

It's my understanding that

• We could remove the push trigger and only keep the release trigger and everything would still work fine.
• Having the two triggers comes in handy when/if one pushes tags via git (for testing purposes). As far as I can remember, @keewis was using the push event (via tags) to test the early versions of the workflow.

The general proposal probably needs some socializing and refinement before we start applying it more broadly. I'd be keen to pursue it if others agree — it could bring some more robustness to our testing suite while also making it simpler.

👍🏽

Here's one example that shows github-activity in action: https://jupyterbook.org/reference/_changelog.html

Glad to hear that releasing is getting easier/simpler :)

There are some smaller tasks we can put into pipelines if we want to squeeze more out there (e.g. updating stable branch, adding contributor lists). The only really necessary tasks are writing the summary, organizing the whatsnew, sending out communications.

I've been thinking about this, and I'm wondering if something like this great tool: github-activity could help us (at least with updating whatsnew.rst). github-activity is able to generate the changelog from the GitHub API. This would eliminate the need for updating whatsnew.rst multiple times (once for each PR). With proper labels, github-activity is able to categorize PRs and Issues during the changelog generation. Another cool feature of github-activity is that it includes other contributions in addition to the default PR submissions: https://github-activity.readthedocs.io/en/latest/#how-does-this-tool-define-contributions-in-the-reports

There's an extreme end of "constantly release new main branches". I don't see anything functionally wrong with that, but I'm sure there are unexplored consequences, and I'm not sure our role is to forge a new path in OSS release cycles.

I'm curious.. How are these "new main branches" different from git tags from releases?

So maybe monthly is a reasonable balance?

👍🏽 for frequent/monthly release cadence

5267 seems to accidentally have disabled the scheduled CI.

Oooops! Thanks for catching that.

@alexamici @aurghs any thoughts?

@alexamici @aurghs, the gentlest of bumps on this. Have a few minutes to take a look at this? :smile:

I expected dimension order not to matter, as it is the case in most of xarray. Right now, my code wraps func in a another function that performs this transpose when needed.

Isn't it the case the order matters here because you are providing a template for map_blocks to follow? Would your code still work if you were to leave map_blocks to do the inference of the output dataset on its own as Deepak pointed out? From the code snippet you provided, it appears that map_blocks works without the template

python In [10]: dac.map_blocks(func).load() Out[10]: <xarray.DataArray <this-array> (y: 3, x: 2)> array([[0, 3], [1, 4], [2, 5]]) Dimensions without coordinates: y, x

I believe I have addressed most, if not all changes requested during the review. If there are still disputable changes that need addressing, please let me know

Shall we also add parts of Stephan's comment to the "Contributing Guide" in the documentation?

👍🏽 for addressing this in a separate PR

@max-sixty, I'm going to address a few issues pointed out during the review, then we can merge afterwards.

@shoyer, Thank you for the thorough feedback.

Even code that has been carefully checked can introduce bugs. Style clean-up PRs thus still need to be reviewed carefully.

Fewer lines or tokens of code is not always better. Good examples from this PR include reduced nesting with early return statements, comprehensions and conditional expressions. All of these would be unobjectionable to use in new code (and are often a good idea), but code that doesn't use them even when it could is fine, too.

👍🏽. I reverted back to the original style in most places. There're a few comments that I haven't addressed yet. I will look into those sometime this weekend.

Code-base wide clean-up for particular issues may be warranted occasionally, but I would suggest always discussing it with other core developers first.

I agree. I didn't expect the PR to get this large. My original intent when I created the PR was to clean the backends code, and I ended up invading other areas :(. In the future, I'll definitely make sure to start a discussion before working on changes that may end up affecting a large chunk of the code base.

maybe we should make the "upload to TestPyPI" a separate step?

Or we could just set

yaml skip_existing: true

on the upload to TestPyPI step.

@keewis, do we need to do anything with the release on test.pypi.org? (since the v0.18 is already published there)

hmm should I also delete the tag?

Yes...

I accidentally closed this via Github's keywords... The bug in the CI workflow should be fixed.

Also, feel free to merge this at your earliest convenience

@TomNicholas & @alexamici... If you still want to use this, you may want to delete the old release/tag and re-publish it :( Sorry for the extra work

Never mind... There's a bug in this if

https://github.com/pydata/xarray/blob/01c1d25578634306b5d3179989a0a9bd867de176/.github/workflows/pypi-release.yaml#L84

It should be github.ref instead of github.event.ref... Opening a PR shortly (not sure if you still want to use the GitHub action)...

Hmm... Looks like it built but didn't upload: pydata/xarray/runs/2520588073?check_suite_focus=true

cc @keewis @andersy005

Sorry about that. I think the culprit here are the if conditions which evaluate to false on release events... @keewis what do you think? :)

https://github.com/pydata/xarray/blob/01c1d25578634306b5d3179989a0a9bd867de176/.github/workflows/pypi-release.yaml#L66

https://github.com/pydata/xarray/blob/01c1d25578634306b5d3179989a0a9bd867de176/.github/workflows/pypi-release.yaml#L75

Thank you both for the feedback! Merging this shortly...

LGTM, assuming this worked properly in this PR itself :)

I think it's working properly. I'm not seeing new CI runs for this PR in my fork

looks good to me. I thought we did that already

As far as I can tell, we did it for the upstream-dev CI only

should we remove the push:branches trigger?

I think so

Duplicate of https://github.com/pydata/xarray/pull/5269

I feel this PR is too big and diverse to accept it as a whole.

For example I like most changes from older style formatting to f-strings, but I disagree strongly the moving the return statements into functions improves readability.

If folks are in favor of the original style (inline variables that are immediately returned) throughout the codebase, I'm happy to revert back to the original style :)

Here's the workflow visualization graph. Let me know if the current job dependency is okay...

Also, someone with admin permissions on PyPI should make sure to get the necessary tokens from PyPI and TestPyPI and set them on this repo.

Thanks, @max-sixty

@andersy005 I'm curious, why do you go with multiple jobs within the workflow, and using artifacts to transfer state between them, rather than multiple steps in a single job?

I have a tendency to split a workflow into multiple jobs because it makes reasoning about the workflow easy (at least for me :)) . However, I think using a single job here would reduce overhead since the logic isn't complex to warrant a need for multiple jobs...