issue_comments

@btickell sorry, I basically haven't used this data since that last post, so I don't recall exactly. But I believe I just was able to download it and work with it locally rather than using opendap.

Thanks @dcherian. Note that I was able to access the subset of this dataset that I need via other means, so this is no longer important for me. So feel free to close if it doesn't seem worth the trouble.

I should add: the file in question is one that somebody else (who is on vacation) works with via xarray, or at least has in the past. So that makes me wonder if this behavior is a regression in 0.16.0.

Just came across this PR while trying for the first time to create an animation of xarray data. Looks like it got quite far along but then sputtered. Did it get superseded by hvplot?

Great, thanks @max-sixty and happy to do it.

OK @max-sixty I've gotten logic that works in the sense that it works for this use case and doesn't break any existing tests. But I could use a look over by somebody more familiar with groupby + multi-indexing, as it's totally possible that this quick fix causes other problems.

Thanks @max-sixty. Contrary to my warning about not doing a PR, I couldn't help myself and dug in a bit. It turns out that string coordinates aren't the problem, it's when the coordinate isn't in sorted order. For example, @chrisroat's original example doesn't error if the coordinate is ["G", "R"] instead of ["R", "G"]. A more concrete WIP test:

```python def test_stack_groupby_unsorted_coord(): data = [[0, 1], [2, 3]] data_flat = [0, 1, 2, 3] dims = ["y", "x"] y_vals = [2, 3]

# "y" coord is in sorted order, and everything works
arr = xr.DataArray(data, dims=dims, coords={"y": y_vals})
actual1 = arr.stack(z=["y", "x"]).groupby("z").first()
midx = pd.MultiIndex.from_product([[2, 3], [0, 1]], names=dims)
expected1 = xr.DataArray(data_flat, dims=["z"], coords={"z": midx})
xr.testing.assert_equal(actual1, expected1)

# Now "y" coord is NOT in sorted order, and the bug appears
arr = xr.DataArray(data, dims=dims, coords={"y": y_vals[::-1]})
actual2 = arr.stack(z=["y", "x"]).groupby("z").first()
midx = pd.MultiIndex.from_product([[3, 2], [0, 1]], names=dims)
expected2 = xr.DataArray(data_flat, dims=["z"], coords={"z": midx})
xr.testing.assert_equal(actual2, expected2)

test_stack_groupby_str_coords() yieldspython

AssertionError Traceback (most recent call last)

[...]

AssertionError: Left and right DataArray objects are not equal

Differing values: L array([2, 3, 0, 1]) R array([0, 1, 2, 3]) Differing coordinates: L * z (z) MultiIndex - z_leve...(z) int64 2 2 3 3 - z_leve...(z) int64 0 1 0 1 R * z (z) MultiIndex - y (z) int64 3 3 2 2 - x (z) int64 0 1 0 1 ```

I'll return to this tomorrow, in the meantime if this triggers any thoughts about the best path forward, that would be much appreciated!

Notice that the string coordinate also gets reordered alphabetically: in @chrisroat 's example above, the coord goes from ['R', 'G'] to ['G', 'R'].

@max-sixty I can't promise a PR anytime soon, but if/when I do manage, where would be a good starting point? Perhaps here where the _level_ names are introduced: https://github.com/pydata/xarray/blob/009aa66620b3437cf0de675013fa7d1ff231963c/xarray/core/dataset.py#L251-L256

Edit: actually maybe here: https://github.com/pydata/xarray/blob/9eec56c833da6dca02c3e6c593586fd201a534a0/xarray/core/variable.py#L2237-L2249

Here's a quick and dirty workaround that works at least for my use case. arr_orig is the original DataArray from which arr_unstacked_bad was generated via a stack/groupby/apply/unstack chain yielding the _level_0 etc. dims, with the stack call having been arr_orig.stack(**{dim_of_stack: dims_stacked}). Likely excessively convoluted and YMMV.

```python def fix_unstacked_dims(arr_unstacked_bad, arr_orig, dim_of_stack, dims_stacked): """Workaround for xarray bug involving stacking str-based coords.

C.f. https://github.com/pydata/xarray/issues/3287

"""
dims_not_stacked = [dim for dim in arr_orig.dims if dim not in dims_stacked]
stacked_dims_after_unstack = [dim for dim in arr_unstacked_bad.dims 
                              if dim not in dims_not_stacked]
dims_mapping = {d1: d2 for d1, d2 in zip(stacked_dims_after_unstack, dims_stacked)}
arr_unstacked_bad = arr_unstacked_bad.rename(dims_mapping)

arr_out = arr_orig.copy(deep=True)
arr_out.values = arr_unstacked_bad.transpose(*arr_orig.dims).values
return arr_out.assign_coords(arr_orig.coords)

```

Same or different problem as https://github.com/pydata/xarray/issues/1483?

I just bumped into this problem as well. xarray 0.15.0. Expected behavior? Bug?

Looks like its back up and running now

Indeed, looks like Azure is having some problem: https://status.dev.azure.com/

Pipelines (in US only) listed as "degraded". I.e. not specific to xarray --- I'm having same problem in another repo

No longer needed; closing

@lvankampenhout just FYI #2191 has been opened for further discussion of adding resample to CFTimeIndex. So keep an eye on that for those developments...as well as consider taking a stab at implementing it yourself! I'm sure @spencerkclark and others will be keen to help out once you (or somebody) gets started.

Closed by #1252 ?

Credit also due to @rabernat for organizing the workshop in late 2016 where this effort got off the ground, and to @shoyer who sketched out an initial roadmap for the implementation at that meeting.

So excited to have this in! In aospy alone, we'll be able to get rid of 100s (1000+?) of lines of code now that CFTime is in place.

Just seeing this. I'm tied up the next couple days but would be happy to review it on Wednesday. Although I doubt I'll find anything you or @shoyer wouldn't, so feel free to merge if you'd rather not hold it up another few days.

FWIW aospy is having similar failures starting roughly at the same time: https://github.com/spencerahill/aospy/issues/238

OMG this is so cool!

ditto wow I can't wait for this to be in!

Try running them via pytest instead (pip install pytest or conda install pytest if you don't have it already).

Note that you'll need to add new test(s) that cover the modifications you have made -- not just run the existing tests.

have not run tests yet

These will definitely be needed. Existing tests for open_mfdataset are in xarray/tests/test_backends.py. @shoyer and/or @jhamman can hopefully help you out if you need more guidance.

Not sure how to install flake8

pip install flake8 should work. Alternatively if you use conda conda install -c conda-forge flake8

Do not think the change is big enough to be added to the files

A brief note in What's New under the Enhancements section would actually be appropriate. Just follow the example of the others in that section.

Thanks for taking this on! We actually just bumped across this (here), so your fix will immediately benefit more than just you.

I think you've accidentally used "minimum" instead of "minimal"

One bit that may be helpful: this is arising in aospy's tests, e.g. here. But, they're only occurring in our 'py27-min' environment (and are doing so with 100% consistency); never in our full 'py27' environment.

environment-py27-min.yml:

name: test_env channels: - conda-forge dependencies: - python=2.7 - scipy - netCDF4 - xarray - dask - distributed - pip: - coveralls - pytest-cov

environment-py27.yml

name: test_env channels: - conda-forge dependencies: - python=2.7 - scipy - netCDF4 - xarray - dask - distributed - pytest - future - matplotlib - ipython - pip: - coveralls - pytest-cov - pytest-catchlog - runipy

any final comments

@jhamman none; thanks @shoyer for including the function-version of where

Thanks @shoyer. I guess what I had in mind is the case where both x and y are scalars, while cond is still a condition on a. In that case you couldn't do x.where(cond, y); it would require either a.where(cond, x, y) or where(cond, x, y) being supported. Am I understanding that correctly? (If I'm not being clear, consider a concrete case by plugging in e.g. x=-2, y=0, and cond=(a.x + a.y < 5).)

a.where(cond, x, y) might seem odd, since it doesn't actually retain any of a's values, but then it could retain coordinates and attributes, so it might still be useful. And this differs from where(cond, x, y), which it seems would retain cond's coords and attrs.

How difficult would it be to include np.where's option to provide values for both where the condition is met and where it isn't? From their docstring:

If both x and y are specified, the output array contains elements of x where condition is True, and elements from y elsewhere.

From your example above (haven't gone through the code), what you have implemented in this PR is a special case, namely the xarray analog to np.where(a.x + a.y < 5, a, -1).

I recently had a usecase where this would be handy.

One other thought for x rather than xr: to me at least, the x-prefixed package names just sound cool 😄

Pinging folks on this. Summer is upon us and, generally speaking for the academics among us, is a good time for projects like this.

@spencerkclark looks like you're still looking for guidance w/r/t the last batch of comments from May 10.

From the CI it also looks like I need to accommodate some updates in pandas in NetCDFTimeIndex. I will try and look into that tomorrow.

Did your subsequent commits resolve this?

See also Dataset.set_coords

open_mfdataset has a 'concat_dim' optional keyword argument where you can specify the name of a new dimension that you want to concatenate your files over. You can read more about this in the API reference on open_mfdataset.

You could then overwrite the coordinate of that new dimension with your desired time coordinate. Does that help?

+1 we have come across this recently also in aospy

Sounds like Dataset.reset_coords is what you're looking for.

I'm not sure we want to wrap np.gradient. It seems like other approaches like @rabernat 's xgcm would be more appropriate as a superset of xarray.

Certainly grid-aware differencing and integral operators are preferred when the grid information is known and available, but I'm not sure that therefore a more naive version akin to np.gradient would not be useful. It's quite likely that there are xarray users (e.g. in non climate/weather/ocean-related fields) wherein a 'c' grid is meaningless to them, yet they still would appreciate being able to easily compute derivatives via xarray operations.

But then we're back to the valid questions raised before re: what is the appropriate scope of xarray functionality, c.f. https://github.com/pydata/xarray/issues/1288#issuecomment-283062107 and subsequent in that thread

I capitalize Xarray when used for titles or at the beginning of a sentence

I agree with this: lower case unless beginning of a sentence or in a title. If you capitalize the "A" too, then there's more of a discrepancy between the uncapitalized version and the capitalized one.

FYI this appears to have just been fixed on the RTD end: https://github.com/rtfd/readthedocs.org/issues/2651#issuecomment-284922193

I like the integrate idea. Nothing further to add not already covered nicely via the above concerns by @rabernat and responses by @shoyer.

@spencerkclark @shoyer I got some more concrete information c.f. on my previous comment on negative and/or 5 digit dates. The TRACE simulation outputs netCDF files uses units of thousands of years relative to 1950, and therefore doesn't use 5 integers. But it does use negative and positive floats...negative for <1950, positive for >1950.

Re: 5 digits, there is growing research interest in very long climate model integrations, e.g. http://www.longrunmip.org/, but even those for now appear <10k yr in duration.

But there are also so called EMICs (Earth Models of Intermediate Complexity) that are cheap to run for 1000s of years. Although I couldn't immediately find any published results using them for >10k yr duration...

So ultimately I'd say there definitely exists a use-case for negative times (albeit with odd format) and there likely exists a use-case for 5 digit years. IMHO these are not must-haves for the initial netcdftime implementation but should at least be kept in mind, i.e. code design that makes them not-overly-difficult to introduce eventually.

I suspect there are xarray users with more direct experience with these cases...feel free to chime in

Sorry this example wasn't clear.

Note that v0.9.1 contains the new drop argument to .isel for exactly these sorts of use cases: ds.isel(dim_0, drop=True) should work regardless of whether or not there is a dim_0 coordinate.

Yes, this looks like the perfect solution for our use-case. My mistake for not reading the docs carefully enough. Thanks!

Great, see #1219.

for now I think it makes sense to keep this in xarray proper

I agree.

This is great. At least in terms of the functionality I was looking for, I'd say this closes #754.

I think your basic example is probably already enough to be useful.

Just to be sure we're not mixing Spencers, this was all @spencerkclark's great work! I had no hand in it.

I think if we were to include string-based indexing, it would be best if it were completely consistent with the DatetimeIndex version.

I agree.

So ultimately this raises the question, would we want to add just the field accessors to enable group-by operations for now and add string-based selection (and other features like resample) later, or should we put our heads down and work out a solution for partial datetime string based using netcdftime datetime objects?

Maybe an interim solution is for NetCDFTimeIndex to only accept full datestrings, issuing an error message for partial strings explaining that this functionality is forthcoming?

This looks pretty sane to me, though of course it's still missing a few nice things you can do with datetime64 (e.g., reindex and partial datetime string selection).

@shoyer and others, is there a well-defined list of required features that the new index object would need to satisfy in order to be considred for inclusion in xarray? Are the two that you mentioned must-haves? Are there others?

Wanting to make sure we're all on the same page in terms of what the target is.

@shoyer maybe just a name change then? cartopy obviously is inherently geo-specific. Is it possible that other fields would find the utilities in geocolormesh useful? Or could the geocolormesh functionality be refactored into the existing plotting framework? (Just thinking out loud...not familiar with the plotting codebase at all)

Pandas has created a poll on their mailing list about this issue...I encourage everybody to speak up there: https://groups.google.com/forum/#!topic/pydata/kk04maBGw1U

(Will blast this to xarray mailing list also)

Thanks for the ping. Sorry, I haven't. I'm actually submitting my PhD thesis in ~2 months and then defending it this summer, so I won't be able to help much until August or September. So by all means, go for it.

Indeed. I'm happy to do this sometime (eventually...), but if somebody ends up with a pressing need for this in the meantime, go for it. Thanks!

Not to be pedantic, but just one more :+1: on ultimately implementing units support within xarray -- that would be huge.

@shoyer thanks! Good to know.

@MaximilianR FWIW I also had problems with .version recently when I cloned from the main repo...ended up having to copy version.py from another xarray install. Maybe has something to do with .gitignore? xarray/version.py is included there.

No problem. Sorry for the extra commit -- I tried to squash them together, but I'm new to rebase and apparently didn't do it right. Thanks for helping me incorporate this!

Sure, will do all of these.

Thanks, to_dataset is a good workaround.

Sure, will do in the coming days or over the weekend. Thanks.

One more vote for import xarray as xr. It mimics import numpy as np, in terms of retaining the first consonant of each syllable.

Also, it is clearly not pronounceable other than as the individual letters sequentially, i.e. as 'ex-are', which is a good thing. xy, xra, and xry all have ambiguous pronunciations.

~~@spencerkclark and I have come across the same problem in one of the atmospheric models we work with, but in our case in addition to the time bounds array the latitude and longitude bounds are also 2D. I mention this because I suspect this wouldn't be resolved by @shoyer 's previous comment re: fixing decode_cf_timedelta.~~ I was wrong here; see comments in #667

@shoyer Sure, just did: #667. ~~My suspicion is that it's ultimately the same underlying issue, which is simply that these bound arrays have a 2nd dimension that xray is having a hard time dealing with. Nevertheless, that it occurs in non-time coordinates as well implies that dealing with it solely through time-related methods won't solve everything.~~ I was wrong here; see comments in #667.

Sorry, @spencerkclark is right, the ValueError issue we had was also due to the 2D time bounds array only. For example: (the netCDF file used below is also at ftp://ftp.gfdl.noaa.gov/pub/s1h/atmos.201001-201012.t_surf.nc)

``` python In [1]: ds = xray.open_dataset('/archive/Spencer.Hill/am3/am3clim_hurrell/gfdl.ncrc2-intel-prod-openmp/pp/atmos/ts/monthly/1yr/atmos.201001-201012.t_surf.nc')

In [2]: print(ds)

ValueError Traceback (most recent call last) <ipython-input-2-4d24098ddece> in <module>() ----> 1 print(ds)

/home/s1h/anaconda/lib/python2.7/site-packages/xray/core/dataset.pyc in repr(self) 885 886 def repr(self): --> 887 return formatting.dataset_repr(self) 888 889 @property

...

/home/s1h/anaconda/lib/python2.7/site-packages/pandas/tseries/timedeltas.pyc in _convert_listlike(arg, box, unit, name) 47 value = arg.astype('timedelta64[{0}]'.format(unit)).astype('timedelta64[ns]', copy=False) 48 else: ---> 49 value = tslib.array_to_timedelta64(_ensure_object(arg), unit=unit, errors=errors) 50 value = value.astype('timedelta64[ns]', copy=False) 51

pandas/tslib.pyx in pandas.tslib.array_to_timedelta64 (pandas/tslib.c:47046)()

ValueError: Buffer has wrong number of dimensions (expected 1, got 2)

In [3]: ds2 = ds.drop('time_bounds')

In [4]: print(ds2) <xray.Dataset> Dimensions: (bnds: 2, lat: 90, lon: 144, time: 12) Coordinates: * lat (lat) float64 -89.0 -87.0 -85.0 -83.0 -81.0 -79.0 -77.0 ... * lon (lon) float64 1.25 3.75 6.25 8.75 11.25 13.75 16.25 18.75 ... * time (time) datetime64[ns] 2010-01-16T12:00:00 2010-02-15 ... * bnds (bnds) int64 0 1 Data variables: average_DT (time) timedelta64[ns] 31 days 28 days 31 days 30 days ... average_T1 (time) datetime64[ns] 2010-01-01 2010-02-01 2010-03-01 ... average_T2 (time) datetime64[ns] 2010-02-01 2010-03-01 2010-04-01 ... lat_bnds (lat, bnds) float64 -90.0 -88.0 -88.0 -86.0 -86.0 -84.0 ... lon_bnds (lon, bnds) float64 0.0 2.5 2.5 5.0 5.0 7.5 7.5 10.0 10.0 ... t_surf (time, lat, lon) float64 245.9 245.9 245.8 245.7 245.7 245.6 ... ... ```

The errors I was thinking of relating to these lat- and lon-bounds were ultimately due to errors in my own code...my mistakes appear to be the unifying theme here! Sorry for the confusion.

Cool, thanks. Not familiar enough w/ the xray source code at this time to contribute directly to the refactor, but for the time being here's a simple 2-liner workaround:

python dlon = arr['lon'].diff('lon') dlon.values = np.diff(arr['lon'])