html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/3731#issuecomment-1234643971,https://api.github.com/repos/pydata/xarray/issues/3731,1234643971,IC_kwDOAMm_X85JlywD,3698640,2022-09-01T18:36:30Z,2022-09-01T18:36:30Z,CONTRIBUTOR,"FWIW I could definitely see use cases for allowing something like this... I have used cumbersome/ugly workarounds to work with variance-covariance matrices etc. So I'm not weighing in on the ""this should raise an error"" debate. I got briefly excited when I saw it *didn't* raise an error, until everything started unraveling 🙃 ","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,557257598
https://github.com/pydata/xarray/issues/3731#issuecomment-1234639080,https://api.github.com/repos/pydata/xarray/issues/3731,1234639080,IC_kwDOAMm_X85Jlxjo,3698640,2022-09-01T18:31:08Z,2022-09-01T18:31:08Z,CONTRIBUTOR,"ooh this is a fun one! came across this issue when we stumbled across a pendantic case writing tests (H/T @brews). I expected this to ""fail loudly in the constructor"" but it doesn't. note that currently AFAICT you cannot use positional slicing to achieve an intuitive result - the behavior seems more undefined/unpredictable
```python
# setup
import xarray as xr, pandas as pd, numpy as np
da = xr.DataArray(np.arange(8).reshape(2, 2, 2), coords=[[0, 1], [0, 1], ['a', 'b']], dims=[""ni"", ""ni"", ""shh""])
```
xarray seems to not know it has a problem:
```python
In [4]: da
Out[4]:
array([[[0, 1],
[2, 3]],
[[4, 5],
[6, 7]]])
Coordinates:
* ni (ni) int64 0 1
* shh (shh)
array([0, 1])
Coordinates:
ni int64 0
* shh (shh)
array([[[0, 1],
[2, 3]],
[[4, 5],
[6, 7]]])
Coordinates:
* ni (ni) int64 0 1
* shh (shh)
array([0, 1])
Coordinates:
ni int64 0
* shh (shh) If there's anything you need help with or would like to discuss, please don't hesitate to raise [a Zarr issue](https://github.com/zarr-developers/zarr-python/issues). We also enabled GH discussions over there so if that fits better feel free to use that 🙂
@jakirkham appreciate that! And just to clarify my confusion/frustration in the past was simply around the issue that I'm documenting here, which I've finally figured out! so hopefully this will help resolve the problem for future users. I agree with Ryan that there might be some changes in the defaults that could be helpful here, though setting intuitive defaults other than zarr's defaults could get messy for complex datasets with a mix of dask and in-memory arrays. But I *think* that's all on the xarray side?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1122796867,https://api.github.com/repos/pydata/xarray/issues/6542,1122796867,IC_kwDOAMm_X85C7IVD,3698640,2022-05-10T19:48:34Z,2022-05-10T19:49:04Z,CONTRIBUTOR,"@jakirkham were you thinking a reference to the dask docs for more info on optimal chunk sizing and aligning with storage? or are you suggesting the proposed docs change is too complex?
I was trying to address the lack of documentation on specifying chunks within a zarr array for *non-dask* arrays/coordinates, but also covering the weedsy (but common) case of datasets with a mix of dask & in-memory arrays/coords like in my example. I have been frustrated by zarr stores I've written with a couple dozen array chunks and thousands of coordinate chunks for this reason, but it's definitely a gnarly topic to cover concisely :P","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1122790681,https://api.github.com/repos/pydata/xarray/issues/6542,1122790681,IC_kwDOAMm_X85C7G0Z,3698640,2022-05-10T19:41:12Z,2022-05-10T19:41:12Z,CONTRIBUTOR,"thank you all for being patient with this PR! seems the build failed again for the same reason. I think there might be something wrong with my examples, though it beats me what the issue is. As far as I can tell, most builds come in somewhere in the mid 900s range on readthedocs but my branch consistently times out at 1900s. I'll see if I can muster a bit of time to run through the exact rtd build workflow and figure out what's going on but probably won't get to it until this weekend.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1119775222,https://api.github.com/repos/pydata/xarray/issues/6542,1119775222,IC_kwDOAMm_X85Cvmn2,3698640,2022-05-06T16:07:44Z,2022-05-06T16:07:44Z,CONTRIBUTOR,"now I'm getting reproducible build timeouts 😭
The docs build alone (just `make html`) takes around an hour on my laptop for xarray/main and just a bit longer on my branch. readthedocs seems to be timing out at 31 minutes. How does this usually work? or am I doing something wrong again?
Thanks for all the help everyone!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1118786888,https://api.github.com/repos/pydata/xarray/issues/6542,1118786888,IC_kwDOAMm_X85Cr1VI,3698640,2022-05-05T16:36:02Z,2022-05-05T16:36:02Z,CONTRIBUTOR,"@dcherian - yep I was trying to follow that guide closely but was still struggling with building using conda on my laptop's miniconda environment. The sphinx ipython directive kept running in the wrong conda environment, even after deleting sphinx, ipython, and ipykernel on my base env and making sure the env I was running `make` from had all the dependencies 🤷🏼♂️ . The docker image helped with the conda version issue but then I ran afoul of the xarray version because I used a shallow clone. oopps 😞 ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1118775756,https://api.github.com/repos/pydata/xarray/issues/6542,1118775756,IC_kwDOAMm_X85CrynM,3698640,2022-05-05T16:31:06Z,2022-05-05T16:31:06Z,CONTRIBUTOR,!!! @andersy005 thank you so much! yes - I was using a shallow clone inside the docker version of the build. I really appreciate the review and for catching my error. I'll clean this up and push the changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1114097888,https://api.github.com/repos/pydata/xarray/issues/6542,1114097888,IC_kwDOAMm_X85CZ8jg,3698640,2022-05-01T01:47:27Z,2022-05-01T01:47:27Z,CONTRIBUTOR,"sorry I know I said I'd fix this but I'm having a very hard time figuring out what is wrong and how to build the docs. I had to set up a docker image to build them because I couldn't get the ipython directive to use the right conda env on my laptop, and now I'm getting a version error when pandas encounters the default build number on my fork. I'm a bit embarrassed that I can't figure this out, but... I think I might need a hand getting this across the finish line 😕
```
....
#13 1597.6 reading sources... [ 97%] getting-started-guide/faq
#13 1600.8 reading sources... [ 97%] getting-started-guide/index
#13 1600.8 reading sources... [ 97%] getting-started-guide/installing
#13 1601.0 reading sources... [ 97%] getting-started-guide/quick-overview
#13 1609.3 WARNING:
#13 1609.3 >>>-------------------------------------------------------------------------
#13 1609.3 Exception in /xarray/doc/getting-started-guide/quick-overview.rst at block ending on line 169
#13 1609.3 Specify :okexcept: as an option in the ipython:: block to suppress this message
#13 1609.3 ---------------------------------------------------------------------------
#13 1609.3 ImportError Traceback (most recent call last)
#13 1609.3 Input In [40], in ()
#13 1609.3 ----> 1 series.to_xarray()
#13 1609.3
#13 1609.3 File /srv/conda/envs/xarray-doc/lib/python3.9/site-packages/pandas/core/generic.py:3173, in NDFrame.to_xarray(self)
#13 1609.3 3096 @final
#13 1609.3 3097 def to_xarray(self):
#13 1609.3 3098 """"""
#13 1609.3 3099 Return an xarray object from the pandas object.
#13 1609.3 3100
#13 1609.3 (...)
#13 1609.3 3171 speed (date, animal) int64 350 18 361 15
#13 1609.3 3172 """"""
#13 1609.3 -> 3173 xarray = import_optional_dependency(""xarray"")
#13 1609.3 3175 if self.ndim == 1:
#13 1609.3 3176 return xarray.DataArray.from_series(self)
#13 1609.3
#13 1609.3 File /srv/conda/envs/xarray-doc/lib/python3.9/site-packages/pandas/compat/_optional.py:164, in import_optional_dependency(name, extra, errors, min_version)
#13 1609.3 162 return None
#13 1609.3 163 elif errors == ""raise"":
#13 1609.3 --> 164 raise ImportError(msg)
#13 1609.3 166 return module
#13 1609.3
#13 1609.3 ImportError: Pandas requires version '0.15.1' or newer of 'xarray' (version '0.1.dev1+gfdf7303' currently installed).
#13 1609.3
#13 1609.3 <<<-------------------------------------------------------------------------
#13 1609.3
#13 1609.3 Exception occurred:
#13 1609.3 File ""/srv/conda/envs/xarray-doc/lib/python3.9/site-packages/IPython/sphinxext/ipython_directive.py"", line 584, in process_input
#13 1609.3 raise RuntimeError('Non Expected exception in `{}` line {}'.format(filename, lineno))
#13 1609.3 RuntimeError: Non Expected exception in `/xarray/doc/getting-started-guide/quick-overview.rst` line 169
#13 1609.3 The full traceback has been saved in /tmp/sphinx-err-a01y480j.log, if you want to report the issue to the developers.
#13 1609.3 Please also report this if it was a user error, so that a better error message can be provided next time.
#13 1609.3 A bug report can be filed in the tracker at . Thanks!
#13 1611.4 make: *** [Makefile:58: html] Error 2
#13 1611.6 ERROR conda.cli.main_run:execute(41): `conda run /bin/bash -c make html` failed. (See above for error)
#13 ERROR: executor failed running [conda run --no-capture-output -n xarray-doc /bin/bash -c make html]: exit code: 2
------
> [10/10] RUN make html:
------
executor failed running [conda run --no-capture-output -n xarray-doc /bin/bash -c make html]: exit code: 2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/pull/6542#issuecomment-1113712207,https://api.github.com/repos/pydata/xarray/issues/6542,1113712207,IC_kwDOAMm_X85CYeZP,3698640,2022-04-29T20:44:43Z,2022-04-29T20:44:43Z,CONTRIBUTOR,hmmm seems I've messed something up in the docs build. apologize for the churn - will fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104
https://github.com/pydata/xarray/issues/6456#issuecomment-1098574761,https://api.github.com/repos/pydata/xarray/issues/6456,1098574761,IC_kwDOAMm_X85Beuup,3698640,2022-04-13T23:34:16Z,2022-04-13T23:34:48Z,CONTRIBUTOR,"> In the example it's saving every iteration, but in my actual code it's much less frequent
when I said ""you're overwriting the file every iteration"" I meant to put the emphasis on _overwiting_. by using `mode='w'` instead of `mode='a'` you're telling zarr to delete the file if it exists and the re-create it every time `to_zarr` is executed.
See the docs on [`xr.Dataset.to_zarr`](https://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_zarr.html):
> **mode** (`{""w"", ""w-"", ""a"", ""r+"", None}`, *optional*) – Persistence mode: “w” means create (overwrite if exists); “w-” means create (fail if exists); “a” means override existing variables (create if does not exist); “r+” means modify existing array values only (raise an error if any metadata or shapes would change). The default mode is “a” if `append_dim` is set. Otherwise, it is “r+” if `region` is set and `w-` otherwise.
This interpretation of mode is consistent across all of python - see the docs for [python builtins: open](https://docs.python.org/3/library/functions.html#open)
So I think changing your writes to `ds3.to_zarr('zarr_bug.zarr', mode='a')` as Max suggested will get you a good part of the way there :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1197117301
https://github.com/pydata/xarray/pull/6467#issuecomment-1096953860,https://api.github.com/repos/pydata/xarray/issues/6467,1096953860,IC_kwDOAMm_X85BYjAE,3698640,2022-04-12T16:39:53Z,2022-04-12T16:39:53Z,CONTRIBUTOR,"oof - debugging CI build process is no fun. good luck, and thanks @max-sixty !!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1199127752
https://github.com/pydata/xarray/issues/6456#issuecomment-1094411214,https://api.github.com/repos/pydata/xarray/issues/6456,1094411214,IC_kwDOAMm_X85BO2PO,3698640,2022-04-10T23:40:49Z,2022-04-10T23:40:49Z,CONTRIBUTOR,"@tbloch1 following up on Max's suggestion - it looks like you might be overwriting the file with every iteration. See the docs on [ds.to_zarr](https://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_zarr.html) - `mode='w'` will overwrite the file while `mode='a'` will append. That said, you still would need your indices to not overlap. How are you distinguishing between the files? is each one a different point in time?
To me, this doesn't seem likely to be a bug, but is more of a usage question. Have you tried asking on stackoverflow with the xarray tag?
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1197117301
https://github.com/pydata/xarray/pull/6467#issuecomment-1094409424,https://api.github.com/repos/pydata/xarray/issues/6467,1094409424,IC_kwDOAMm_X85BO1zQ,3698640,2022-04-10T23:29:38Z,2022-04-10T23:29:38Z,CONTRIBUTOR,hmm. readthedocs failed because of concurrency limits (my bad) but seems to have failed to automatically retry. can someone give it a nudge?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1199127752
https://github.com/pydata/xarray/pull/6467#issuecomment-1094342045,https://api.github.com/repos/pydata/xarray/issues/6467,1094342045,IC_kwDOAMm_X85BOlWd,3698640,2022-04-10T18:26:32Z,2022-04-10T18:26:32Z,CONTRIBUTOR,"thanks @max-sixty! I was mostly trying to be overly-cautious because the ValueError was so clearly raised intentionally, and I can't figure out why it was necessary. But maybe it was just left over from a time when providing both arguments really would have caused a problem, and it's no longer the case.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1199127752
https://github.com/pydata/xarray/issues/6036#issuecomment-1005162696,https://api.github.com/repos/pydata/xarray/issues/6036,1005162696,IC_kwDOAMm_X8476ZDI,3698640,2022-01-04T20:53:36Z,2022-01-04T20:54:13Z,CONTRIBUTOR,"This isn't a fix for the overhead required to manage an arbitrarily large graph, but note that creating chunks this small (size 1 in this case) is explicitly not recommended. See the dask docs on [Array Best Practices: Select a good chunk size](https://docs.dask.org/en/stable/array-best-practices.html#select-a-good-chunk-size) - they recommend chunks *no smaller than 100 MB*. Your chunks are 8 bytes. This creates 1 billion tasks, which does result in an enormous overhead - there's no way around this. Note that storing this on disk would not help - the problem results from the fact that 1 billion tasks will almost certainly overwhelm any dask scheduler. The general [dask best practices](https://docs.dask.org/en/stable/array-best-practices.html) guide recommends keeping the number of tasks below 1 million if possible.
Also, I don't think that the issue here is in specifying the universe of the tasks that need to be created, but rather in creating and managing the python task objects themselves. So pre-computing or storing them wouldn't help.
For me, changing to (1000, 1000, 100) chunks (~750MB for a float64 array) reduces the time to a couple ms:
```python
In [16]: %%timeit
...:
...: chunks = (1000, 1000, 100)
...: ds = xr.Dataset(data_vars={
...: ""foo"": (('x', 'y', 'z'), dask.array.empty((1000, 1000, 1000), chunks=(1000, 1000, 1000)))})
...: ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}, mode='w')
...: ds_loaded = xr.open_zarr(group='ds.zarr', store='data')
...:
...:
6.36 ms ± 111 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
With this chunking scheme, you could store and work with much, much more data. In fact, scaling the size of your example by 3 orders of magnitude only increases the runtime by ~5x:
```python
In [18]: %%timeit
...:
...: chunks = (1000, 1000, 100, 1)
...: ds = xr.Dataset(data_vars={
...: ""foo"": (('w', 'x', 'y', 'z'), dask.array.empty((1000, 1000, 1000, 1000), chunks=(1000, 1000, 1000, 1)))})
...: ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}, mode='w')
...: ds_loaded = xr.open_zarr(group='ds.zarr', store='data')
...:
...:
36.9 ms ± 2.23 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
So if re-writing your arrays with larger chunks is an option I think this could get around the problem you're seeing?","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1068225524
https://github.com/pydata/xarray/issues/5994#issuecomment-1003460778,https://api.github.com/repos/pydata/xarray/issues/5994,1003460778,IC_kwDOAMm_X847z5iq,3698640,2021-12-31T22:36:04Z,2021-12-31T22:41:51Z,CONTRIBUTOR,"This looks like it could be a good improvement! Just flagging that in [`func_interpolate_na` on main](https://github.com/pydata/xarray/blob/main/xarray/core/missing.py#L379-L396) I see a different condition:
```python
# fast track for no-nans and all-nans cases
n_nans = nans.sum()
if n_nans == 0 or n_nans == len(y):
return y
```
Do you see the `n_nans < 0` condition somewhere or is this just a typo? That condition also wouldn't make much sense.
Your point about using the shortcut for `n_nans == len(y) - 1` seems right to me, and definitely doesn't seem worthy of an error when an all-NaN slice is silently skipped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1055867960
https://github.com/pydata/xarray/issues/6124#issuecomment-1002777049,https://api.github.com/repos/pydata/xarray/issues/6124,1002777049,IC_kwDOAMm_X847xSnZ,3698640,2021-12-29T21:09:12Z,2021-12-29T21:09:12Z,CONTRIBUTOR,"I realize this may be a larger discussion, but the implementation is so easy I went ahead and filed a PR that issues a PendingDeprecationWarning in `Dataset.__bool__`.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,1090229430
https://github.com/pydata/xarray/issues/6124#issuecomment-1002673070,https://api.github.com/repos/pydata/xarray/issues/6124,1002673070,IC_kwDOAMm_X847w5Ou,3698640,2021-12-29T16:22:10Z,2021-12-29T16:22:10Z,CONTRIBUTOR,"> If there's a synthesis of keeping the truthiness while reducing the chance of these mistakes, that would be very welcome.
@max-sixty im not sure what this would look like. Do you mean a warning or are you hinting that the bar that would need to be met is a silver bullet that preserves bool(ds) but somehow isn’t confusing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1090229430
https://github.com/pydata/xarray/issues/6124#issuecomment-1002671461,https://api.github.com/repos/pydata/xarray/issues/6124,1002671461,IC_kwDOAMm_X847w41l,3698640,2021-12-29T16:18:35Z,2021-12-29T16:18:35Z,CONTRIBUTOR,"Yeah… I do understand how it’s currently working and why, and the behavior is certainly intuitive to those who appreciate the mapping inheritance.
That said, I feel I have to make a last stand argument because this trips people up quite often (on my team and elsewhere). I haven’t yet come across an example of anyone using this correctly, but I see users misusing it all the time.
The examples and behavior you’re showing @Illviljan seem to me like more the natural result of an implementation detail than a critical principle of the dataset design. While it’s obvious why `bool(ds)` resolves to True or False in the current implementation, I still think it’s not used and is out of step with how most users think of a dataset. The fact that nothing in the package requires a change in order to pass tests (the broken tests and the pathological case aside) reaffirms for me that this is not a useful feature.
I don’t know much about the mapping protocol or how closely it must be followed. Is the idea here that packages building on xarray (or interoperability features in e.g. numpy or dask) depend on a strict adherence to the full spec? ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1090229430
https://github.com/pydata/xarray/issues/3951#issuecomment-610615488,https://api.github.com/repos/pydata/xarray/issues/3951,610615488,MDEyOklzc3VlQ29tbWVudDYxMDYxNTQ4OA==,3698640,2020-04-07T20:55:11Z,2020-04-07T20:55:11Z,CONTRIBUTOR,"Here's a test script I'm using for this:
```bash
echo '$ conda create -n py37xr14 -c conda-forge --yes python=3.7 xarray=0.14.1'
conda create -n py37xr14 -c conda-forge --yes python=3.7 xarray=0.14.1 > /dev/null
echo '$ conda create -n py37xr15 -c conda-forge --yes python=3.7 xarray=0.15.1'
conda create -n py37xr15 -c conda-forge --yes python=3.7 xarray=0.15.1 > /dev/null
echo '$ conda run -n py37xr14 python test.py'
conda run -n py37xr14 python test.py
echo
echo '$ conda run -n py37xr15 python test.py'
conda run -n py37xr15 python test.py
echo
echo '$ conda list -n py37xr14'
conda list -n py37xr14
echo
echo '$ conda list -n py37xr15'
conda list -n py37xr15
conda env remove -n py37xr14 > /dev/null 2>&1
conda env remove -n py37xr15 > /dev/null 2>&1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,596115014
https://github.com/pydata/xarray/issues/3951#issuecomment-610612332,https://api.github.com/repos/pydata/xarray/issues/3951,610612332,MDEyOklzc3VlQ29tbWVudDYxMDYxMjMzMg==,3698640,2020-04-07T20:47:51Z,2020-04-07T20:47:51Z,CONTRIBUTOR,yeah I use this pattern all the time - df.stack().to_xarray() seems to now fail unless your columns were sorted alphabetically. not sure yet where this is happening but it does result in some pernicious bad data errors that can be hard to debug if you catch them at all.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,596115014
https://github.com/pydata/xarray/issues/1075#issuecomment-373541528,https://api.github.com/repos/pydata/xarray/issues/1075,373541528,MDEyOklzc3VlQ29tbWVudDM3MzU0MTUyOA==,3698640,2018-03-15T22:21:51Z,2018-03-16T01:33:24Z,CONTRIBUTOR,"xarray==0.10.2
netCDF4==1.3.1
Just tried it again and didn't have any issues:
```python
patt = (
'http://nasanex.s3.amazonaws.com/NEX-GDDP/BCSD/{scen}/day/atmos/{var}/' +
'r1i1p1/v1.0/{var}_day_BCSD_{scen}_r1i1p1_{model}_{year}.nc')
def open_url_dataset(url):
fname = os.path.splitext(os.path.basename(url))[0]
res = requests.get(url)
content = io.BytesIO(res.content)
nc4_ds = netCDF4.Dataset(fname, memory=res.content)
store = xr.backends.NetCDF4DataStore(nc4_ds)
ds = xr.open_dataset(store)
return ds
ds = open_url_dataset(url=patt.format(
model='GFDL-ESM2G', scen='historical', var='tasmax', year=1988))
ds
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655
https://github.com/pydata/xarray/issues/1075#issuecomment-357125148,https://api.github.com/repos/pydata/xarray/issues/1075,357125148,MDEyOklzc3VlQ29tbWVudDM1NzEyNTE0OA==,3698640,2018-01-12T02:27:27Z,2018-01-12T02:27:27Z,CONTRIBUTOR,"yes! Thanks @jhamman and @shoyer. I hadn't tried it yet, but just did. worked great!
```python
In [1]: import xarray as xr
...: import requests
...: import netCDF4
...:
...: %matplotlib inline
In [2]: res = requests.get(
...: 'http://nasanex.s3.amazonaws.com/NEX-GDDP/BCSD/rcp45/day/atmos/tasmin/' +
...: 'r1i1p1/v1.0/tasmin_day_BCSD_rcp45_r1i1p1_CESM1-BGC_2073.nc')
In [3]: res.status_code
Out [3]: 200
In [4]: res.headers['content-type']
Out [4]: 'application/x-netcdf'
In [5]: nc4_ds = netCDF4.Dataset('tasmin_day_BCSD_rcp45_r1i1p1_CESM1-BGC_2073', memory=res.content)
In [6]: store = xr.backends.NetCDF4DataStore(nc4_ds)
In [7]: ds = xr.open_dataset(store)
In [8]: ds.tasmin.isel(time=0).plot()
/global/home/users/mdelgado/git/public/xarray/xarray/plot/utils.py:51: FutureWarning: 'pandas.tseries.converter.register' has been moved and renamed to 'pandas.plotting.register_matplotlib_converters'.
converter.register()
Out [8]:
```

```python
In [9]: ds
Out [9]:
Dimensions: (lat: 720, lon: 1440, time: 365)
Coordinates:
* time (time) datetime64[ns] 2073-01-01T12:00:00 2073-01-02T12:00:00 ...
* lat (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ...
* lon (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ...
Data variables:
tasmin (time, lat, lon) float64 ...
Attributes:
parent_experiment: historical
parent_experiment_id: historical
parent_experiment_rip: r1i1p1
Conventions: CF-1.4
institution: NASA Earth Exchange, NASA Ames Research C...
institute_id: NASA-Ames
realm: atmos
modeling_realm: atmos
version: 1.0
downscalingModel: BCSD
experiment_id: rcp45
frequency: day
realization: 1
initialization_method: 1
physics_version: 1
tracking_id: 1865ff49-b20c-4268-852a-a9503efec72c
driving_data_tracking_ids: N/A
driving_model_ensemble_member: r1i1p1
driving_experiment_name: historical
driving_experiment: historical
model_id: BCSD
references: BCSD method: Thrasher et al., 2012, Hydro...
DOI: http://dx.doi.org/10.7292/W0MW2F2G
experiment: RCP4.5
title: CESM1-BGC global downscaled NEX CMIP5 Cli...
contact: Dr. Rama Nemani: rama.nemani@nasa.gov, Dr...
disclaimer: This data is considered provisional and s...
resolution_id: 0.25 degree
project_id: NEXGDDP
table_id: Table day (12 November 2010)
source: BCSD 2014
creation_date: 2015-01-07T19:18:31Z
forcing: N/A
product: output
```
","{""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655
https://github.com/pydata/xarray/pull/1802#issuecomment-354469859,https://api.github.com/repos/pydata/xarray/issues/1802,354469859,MDEyOklzc3VlQ29tbWVudDM1NDQ2OTg1OQ==,3698640,2017-12-29T16:45:37Z,2017-12-29T16:45:37Z,CONTRIBUTOR,Ok this is good to go if you all do want to enable _FillValue for variable-length unicode strings with a netCDF4 backend. Seems like there's a lot of prior work/thinking in this space though so no worries if you want to wait.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,285006452
https://github.com/pydata/xarray/pull/1802#issuecomment-354379246,https://api.github.com/repos/pydata/xarray/issues/1802,354379246,MDEyOklzc3VlQ29tbWVudDM1NDM3OTI0Ng==,3698640,2017-12-29T00:22:57Z,2017-12-29T00:22:57Z,CONTRIBUTOR,lol. no I'm just walking around in your footsteps @shoyer. I've just enabled the tests you presumably wrote for #1647 & #1648. Curious why variable-length unicode strings with _FillHoles using netCDF4 doesn't currently work in master?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,285006452
https://github.com/pydata/xarray/pull/1802#issuecomment-354378158,https://api.github.com/repos/pydata/xarray/issues/1802,354378158,MDEyOklzc3VlQ29tbWVudDM1NDM3ODE1OA==,3698640,2017-12-29T00:11:53Z,2017-12-29T00:11:53Z,CONTRIBUTOR,"hmm. Seems I'm touching on a much larger issue here: Unidata/netcdf4-python#730
The round-trip works for me using a netcdf4 engine once this fix is implemented in conventions.py. There are tests that are ready to demonstrate this in test_backends.py:836-843, but running these tests (by removing the `pytest.raises` lines) applies to both netCDF4 and h5netcdf backends.
Should these use cases be split up?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,285006452
https://github.com/pydata/xarray/issues/1781#issuecomment-354370636,https://api.github.com/repos/pydata/xarray/issues/1781,354370636,MDEyOklzc3VlQ29tbWVudDM1NDM3MDYzNg==,3698640,2017-12-28T22:55:28Z,2017-12-28T22:55:28Z,CONTRIBUTOR,"I've got a reproducible example of this (sorry for the length):
```python
In [1]: import xarray as xr
...: import numpy as np
...: import pandas as pd
...: import netCDF4
```
```python
In [2]: xr.show_versions()
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.3.0
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.21.0
numpy: 1.13.3
scipy: 1.0.0
netCDF4: 1.3.1
h5netcdf: None
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.0
matplotlib: 2.1.0
cartopy: None
seaborn: 0.8.1
setuptools: 36.5.0.post20170921
pip: 9.0.1
conda: None
pytest: 3.3.1
IPython: 6.2.1
sphinx: None
```python
In [3]: nds = netCDF4.Dataset('test.nc', 'w', format='NETCDF4')
```
```python
In [4]: nds.createDimension('X', 3)
...: nds.createVariable('X', str, ('X', ), fill_value='invalid')
...: nds.createVariable('xname', str, ('X', ), fill_value='invalid')
...: nds.createVariable('var1', np.float32, ('X', ), fill_value=np.nan)
...:
...: nds.variables['X'][:] = np.array(['first', 'second', 'third'])
...: nds.variables['xname'][:] = np.array(['the_first', 'invalid', 'the_third'])
...: nds.variables['var1'][:] = np.random.random((3, ))
```
```python
In [5]: nds.variables['X']
```
vlen X(X)
_FillValue: invalid
vlen data type:
unlimited dimensions:
current shape = (3,)
```python
In [6]: nds.variables['X'][:]
```
array(['first', 'second', 'third'], dtype=object)
```python
In [7]: nds.variables['xname']
```
vlen xname(X)
_FillValue: invalid
vlen data type:
unlimited dimensions:
current shape = (3,)
```python
In [8]: nds.variables['xname'][:]
```
array(['the_first', 'invalid', 'the_third'], dtype=object)
```python
In [9]: nds.variables['var1']
```
float32 var1(X)
_FillValue: nan
unlimited dimensions:
current shape = (3,)
filling on
```python
In [10]: nds.variables['var1'][:]
```
array([ 0.58315825, 0.82443351, 0.39345944], dtype=float32)
```python
In [11]: nds.close()
```
```python
In [12]: ds = xr.open_dataset('test.nc')
```
---------------------------------------------------------------------------
UnboundLocalError Traceback (most recent call last)
in ()
----> 1 ds = xr.open_dataset('test.nc')
~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)
303 lock = _default_lock(filename_or_obj, engine)
304 with close_on_error(store):
--> 305 return maybe_decode_store(store, lock)
306 else:
307 if engine is not None and engine != 'scipy':
~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/backends/api.py in maybe_decode_store(store, lock)
223 store, mask_and_scale=mask_and_scale, decode_times=decode_times,
224 concat_characters=concat_characters, decode_coords=decode_coords,
--> 225 drop_variables=drop_variables)
226
227 _protect_dataset_variables_inplace(ds, cache)
~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables)
1153 vars, attrs, coord_names = decode_cf_variables(
1154 vars, attrs, concat_characters, mask_and_scale, decode_times,
-> 1155 decode_coords, drop_variables=drop_variables)
1156 ds = Dataset(vars, attrs=attrs)
1157 ds = ds.set_coords(coord_names.union(extra_coords).intersection(vars))
~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables)
1086 k, v, concat_characters=concat_characters,
1087 mask_and_scale=mask_and_scale, decode_times=decode_times,
-> 1088 stack_char_dim=stack_char_dim)
1089 if decode_coords:
1090 var_attrs = new_vars[k].attrs
~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim)
998 if (has_fill or scale_factor is not None or add_offset is not None):
999 if has_fill and np.array(fill_value).dtype.kind in ['U', 'S', 'O']:
-> 1000 if string_encoding is not None:
1001 raise NotImplementedError(
1002 'variable %r has a _FillValue specified, but '
UnboundLocalError: local variable 'string_encoding' referenced before assignment
This seems to be produced by the fact that `string_encoding` is not defined for unicode-encoded variable-length string types. Could the problem be the lack of `'O'` and `'U'` in conventions.py line [952](https://github.com/pydata/xarray/blob/6eac8574f85623b18b59ba1ac9cf95e09c87980b/xarray/conventions.py#L952)? I'll try my hand at a PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282061228
https://github.com/pydata/xarray/issues/1075#issuecomment-347705483,https://api.github.com/repos/pydata/xarray/issues/1075,347705483,MDEyOklzc3VlQ29tbWVudDM0NzcwNTQ4Mw==,3698640,2017-11-28T23:58:41Z,2017-11-28T23:58:41Z,CONTRIBUTOR,"Thanks @shoyer. So you can download the entire object into memory and then create a file image and read that? While not a full fix, it's definitely an improvement over download-to-disk-then-read workflow!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655
https://github.com/pydata/xarray/issues/1472#issuecomment-314154076,https://api.github.com/repos/pydata/xarray/issues/1472,314154076,MDEyOklzc3VlQ29tbWVudDMxNDE1NDA3Ng==,3698640,2017-07-10T16:08:30Z,2017-07-10T16:08:30Z,CONTRIBUTOR,"I wasn't misled by the docs, just by my intuition. But now that you've made the distinction between `'2012'` and `'2012-12-31'` this totally makes sense:
```python
In [24]: arr.sel(time='2012')
Out[24]:
array([[ 0.244146, 0.702819, 0.06614 , 0.211758]])
Coordinates:
* age (age) object 'age0' 'age1' 'age2' 'age3'
* time (time) datetime64[ns] 2012-12-31
In [25]: arr.sel(time='2012-12-31')
Out[25]:
array([ 0.244146, 0.702819, 0.06614 , 0.211758])
Coordinates:
* age (age) object 'age0' 'age1' 'age2' 'age3'
time datetime64[ns] 2012-12-31
In [26]: arr.sel(time=np.datetime64('2012-12-31'))
Out[26]:
array([ 0.244146, 0.702819, 0.06614 , 0.211758])
Coordinates:
* age (age) object 'age0' 'age1' 'age2' 'age3'
time datetime64[ns] 2012-12-31
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,241389297
https://github.com/pydata/xarray/issues/1472#issuecomment-314136206,https://api.github.com/repos/pydata/xarray/issues/1472,314136206,MDEyOklzc3VlQ29tbWVudDMxNDEzNjIwNg==,3698640,2017-07-10T15:10:50Z,2017-07-10T15:10:50Z,CONTRIBUTOR,thanks for the schooling. learned something new!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,241389297
https://github.com/pydata/xarray/issues/1075#issuecomment-258025809,https://api.github.com/repos/pydata/xarray/issues/1075,258025809,MDEyOklzc3VlQ29tbWVudDI1ODAyNTgwOQ==,3698640,2016-11-02T23:03:34Z,2016-11-02T23:03:34Z,CONTRIBUTOR,"Got it. :( Thanks!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655
https://github.com/pydata/xarray/issues/798#issuecomment-257062571,https://api.github.com/repos/pydata/xarray/issues/798,257062571,MDEyOklzc3VlQ29tbWVudDI1NzA2MjU3MQ==,3698640,2016-10-29T01:26:47Z,2016-10-29T01:26:47Z,CONTRIBUTOR,"Could this extend the OpeNDAP interface? That would solve the metadata problem and would provide quick access to the distributed workers.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,142498006
|