issue_comments

LGTM :+1:

should we merge this before releasing 0.16.1?

@kmuehlbauer, I modified concat so #4072 and #2811 might already be fixed. Can you confirm?

Edit: at least #4072 does not seem to be fixed yet. ~Good news is that this doesn't seem to depend on a set (i.e. the result does not change with different interpreter sessions)~

@keewis Unfortunately your changes did not help solving #2811. I'll have a pull request ready today with a proposal how to fix both issues.

@keewis See #4419

thanks for checking!

@keewis I'm traveling currently, will check on Monday and come back to you.

@keewis @dcherian FYI: Regarding #4072 and #2811 with concat I successfully tested a fix locally yesterday. I'll ping you when I have the PR out.

the tests pass so this should be ready for review

"mix up" in this case means that the order is random (i.e. a set was involved). Sorry if that was unclear.

What do you mean by "mix up"? Also see #2811

thanks, that almost fixed the doctests. However, concat seems to somehow also mix up the coordinates / dimensions. I tried to fix that by not iterating over a set, but I'm totally not sure if there's a better fix.

One way to fix this is to iterate over variables instead of _coord_names, e.g., instead of: python for k in self._coord_names: ... use: python for k in self._variables: if k in self._coord_names: ...

I believe we already use this trick in a few places for exactly this reason.

unfortunately, that's pretty difficult: the set in question is Dataset._coord_names, and there would be a lot to rewrite if we tried to change that to a list. Just for reference, the method where the reordering happens is Dataset._copy_listed.

OK, my guess is that this is happening because there is someplace where we iterate over a Python set (which has a non-deterministic order based on hashing) rather than a Python dict. If we can nail down where that is happening we should remove it!

The plotting thing could be fixed relatively easily here: https://github.com/pydata/xarray/blob/23dc2fc9f2785c348ff821bf2da61dfa2206d283/xarray/core/dataarray.py#L2805-L2832 and the cor5responding function in Dataset.

yes, sorry, you're right, that is indeed non-deterministic, and exactly the problem I'm having with the doctests.

Interestingly, this seems to be deterministic per python session: running ```python import xarray as xr

ds = xr.Dataset( {"a": (("x", "y"), [[0, 1], [2, 3]])}, coords={"x": ["a", "b"], "y": [0, 1]} ) print(ds) print(ds + 1) print(ds) print(ds + 1) `` will print the same result fordsandds + 1`, but when rerunning in a new session, the coordinate order may be changed (more coordinates increase the probability for the change).

I disagree that this is deterministic. If I run the script multiple times, the plot title varies, and I consider the plot title part of the output.

I have jupyter notebooks that create figures and use this code idiom. If I refactor code of mine that is used by these notebooks, I would like to rerun the notebooks to confirm that the notebook results don't change. Having the plot titles change at random complicates this comparison.

I think sorting the coordinates would avoid this difficulty that I encounter.

I think that's still deterministic (although I agree that the change in the title is annoying): if you run your script multiple times, it will always return the same output. The issue I'm trying to fix here is much worse: the output of the same code will return different results with each run.

Edit: but maybe I just have a messed up environment

Can you give an example of a non-deterministic coordinate order

sure, just try running python -m pytest --doctest-modules xarray --ignore xarray/tests. For me at least (with the environment created by py38.yml), xarray.core.common.DataWithCoords.pipe, xarray.core.groupby.GroupBy.quantile and xarray.core.dataset.Dataset.filter_by_attrs pass or fail at random: in one run they pass, in the next they fail because a different ordering was used for the coordinates (see also #4408).

I think users find it a little more intuitive to preserve the order of dict keys.

Not sure if that's evidence against this (maybe we need to rearrange the repr to really fix it), but in #4072 someone confused the order of the printed coordinates with the order of the dimensions. I'd think it's much easier to tell people that these are alphabetically sorted, no matter what they do, so they shouldn't try to read the dimension order from the coordinates (same with the dimension summary).

Here's another example that yields non-deterministic coordinate order, which propagates into a plot title when selection is done on the coordinates. When I run the code below, the title is sometimes x = 0.0, y = 0.0 and sometimes y = 0.0, x = 0.0.

This is in a new conda environment that I created using the command conda create -n title_order python=3.7 matplotlib xarray. Output from xr.show_versions() is below.

I think the non-determinism is coming from the command ds_subset = ds[['var']].

``` import numpy as np import xarray as xr

xlen = 3 ylen = 4 zlen = 5

x = xr.DataArray(np.linspace(0.0, 1.0, xlen), dims=('x')) y = xr.DataArray(np.linspace(0.0, 1.0, ylen), dims=('y')) z = xr.DataArray(np.linspace(0.0, 1.0, zlen), dims=('z'))

vals = np.arange(xlenylenzlen, dtype='float64').reshape((xlen, ylen, zlen)) da = xr.DataArray(vals, dims=('x', 'y', 'z'), coords={'x': x, 'y': y, 'z': z})

ds = xr.Dataset({'var': da}) print('coords for var in original Dataset') print(ds['var'].coords)

print('****')

ds_subset = ds[['var']] print('coords for var after subsetting') print(ds_subset['var'].coords)

print('****')

p = ds_subset['var'].isel(x=0,y=0).plot() print('title for plot() with dim selection') print(p[0].axes.get_title()) ```

Can you give an example of a non-deterministic coordinate order? That sounds surprising to me, given that on Python 3.6+ dictionaries preserve insertion order.

My preference would be not to sort mappings automatically, either in __init__ or when formatting. I think users find it a little more intuitive to preserve the order of dict keys.

It's true that this is only a guarantee on Python 3.7+, but both CPython 3.6 and all versions of pypy 3 preserve dict insertion order, so in practice we can pretty much always guarantee this. (And soon, Python 3.7 will be required for xarray.)

sorting in __init__ should be possible and might even be easier than sorting when formatting.

Though I was thinking we wouldn't sort it; we'd take the order as given

Do we have to bump the supported python version for that, or is it fine to keep the current behavior for a non-CPython python<3.7 (if we ever supported a language other than CPython)?

I reckon it's fine, it's such a corner case, and it'll only affect these doctests

thanks for the hint, sorting in __init__ should be possible and might even be easier than sorting when formatting. However, 3.7 is the first version of python (the language) which includes the insertion-order preservation introduced in CPython 3.6. Do we have to bump the supported python version for that, or is it fine to keep the current behavior for a non-CPython python<3.7 (if we ever supported a language other than CPython)?

Edit: that breaks two of the iris tests, which seem to depend on the order of the coords. Maybe we should change that?

Could we keep them in sorted order like a normal dict (at least as-of recent python versions)?