html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/3731#issuecomment-1234643971,https://api.github.com/repos/pydata/xarray/issues/3731,1234643971,IC_kwDOAMm_X85JlywD,3698640,2022-09-01T18:36:30Z,2022-09-01T18:36:30Z,CONTRIBUTOR,"FWIW I could definitely see use cases for allowing something like this... I have used cumbersome/ugly workarounds to work with variance-covariance matrices etc. So I'm not weighing in on the ""this should raise an error"" debate. I got briefly excited when I saw it *didn't* raise an error, until everything started unraveling 🙃 ","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,557257598 https://github.com/pydata/xarray/issues/3731#issuecomment-1234639080,https://api.github.com/repos/pydata/xarray/issues/3731,1234639080,IC_kwDOAMm_X85Jlxjo,3698640,2022-09-01T18:31:08Z,2022-09-01T18:31:08Z,CONTRIBUTOR,"ooh this is a fun one! came across this issue when we stumbled across a pendantic case writing tests (H/T @brews). I expected this to ""fail loudly in the constructor"" but it doesn't. note that currently AFAICT you cannot use positional slicing to achieve an intuitive result - the behavior seems more undefined/unpredictable ```python # setup import xarray as xr, pandas as pd, numpy as np da = xr.DataArray(np.arange(8).reshape(2, 2, 2), coords=[[0, 1], [0, 1], ['a', 'b']], dims=[""ni"", ""ni"", ""shh""]) ``` xarray seems to not know it has a problem: ```python In [4]: da Out[4]: array([[[0, 1], [2, 3]], [[4, 5], [6, 7]]]) Coordinates: * ni (ni) int64 0 1 * shh (shh) array([0, 1]) Coordinates: ni int64 0 * shh (shh) array([[[0, 1], [2, 3]], [[4, 5], [6, 7]]]) Coordinates: * ni (ni) int64 0 1 * shh (shh) array([0, 1]) Coordinates: ni int64 0 * shh (shh) If there's anything you need help with or would like to discuss, please don't hesitate to raise [a Zarr issue](https://github.com/zarr-developers/zarr-python/issues). We also enabled GH discussions over there so if that fits better feel free to use that 🙂 @jakirkham appreciate that! And just to clarify my confusion/frustration in the past was simply around the issue that I'm documenting here, which I've finally figured out! so hopefully this will help resolve the problem for future users. I agree with Ryan that there might be some changes in the defaults that could be helpful here, though setting intuitive defaults other than zarr's defaults could get messy for complex datasets with a mix of dask and in-memory arrays. But I *think* that's all on the xarray side?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1122796867,https://api.github.com/repos/pydata/xarray/issues/6542,1122796867,IC_kwDOAMm_X85C7IVD,3698640,2022-05-10T19:48:34Z,2022-05-10T19:49:04Z,CONTRIBUTOR,"@jakirkham were you thinking a reference to the dask docs for more info on optimal chunk sizing and aligning with storage? or are you suggesting the proposed docs change is too complex? I was trying to address the lack of documentation on specifying chunks within a zarr array for *non-dask* arrays/coordinates, but also covering the weedsy (but common) case of datasets with a mix of dask & in-memory arrays/coords like in my example. I have been frustrated by zarr stores I've written with a couple dozen array chunks and thousands of coordinate chunks for this reason, but it's definitely a gnarly topic to cover concisely :P","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1122790681,https://api.github.com/repos/pydata/xarray/issues/6542,1122790681,IC_kwDOAMm_X85C7G0Z,3698640,2022-05-10T19:41:12Z,2022-05-10T19:41:12Z,CONTRIBUTOR,"thank you all for being patient with this PR! seems the build failed again for the same reason. I think there might be something wrong with my examples, though it beats me what the issue is. As far as I can tell, most builds come in somewhere in the mid 900s range on readthedocs but my branch consistently times out at 1900s. I'll see if I can muster a bit of time to run through the exact rtd build workflow and figure out what's going on but probably won't get to it until this weekend.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1119775222,https://api.github.com/repos/pydata/xarray/issues/6542,1119775222,IC_kwDOAMm_X85Cvmn2,3698640,2022-05-06T16:07:44Z,2022-05-06T16:07:44Z,CONTRIBUTOR,"now I'm getting reproducible build timeouts 😭 The docs build alone (just `make html`) takes around an hour on my laptop for xarray/main and just a bit longer on my branch. readthedocs seems to be timing out at 31 minutes. How does this usually work? or am I doing something wrong again? Thanks for all the help everyone!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1118786888,https://api.github.com/repos/pydata/xarray/issues/6542,1118786888,IC_kwDOAMm_X85Cr1VI,3698640,2022-05-05T16:36:02Z,2022-05-05T16:36:02Z,CONTRIBUTOR,"@dcherian - yep I was trying to follow that guide closely but was still struggling with building using conda on my laptop's miniconda environment. The sphinx ipython directive kept running in the wrong conda environment, even after deleting sphinx, ipython, and ipykernel on my base env and making sure the env I was running `make` from had all the dependencies 🤷🏼‍♂️ . The docker image helped with the conda version issue but then I ran afoul of the xarray version because I used a shallow clone. oopps 😞 ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1118775756,https://api.github.com/repos/pydata/xarray/issues/6542,1118775756,IC_kwDOAMm_X85CrynM,3698640,2022-05-05T16:31:06Z,2022-05-05T16:31:06Z,CONTRIBUTOR,!!! @andersy005 thank you so much! yes - I was using a shallow clone inside the docker version of the build. I really appreciate the review and for catching my error. I'll clean this up and push the changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1114097888,https://api.github.com/repos/pydata/xarray/issues/6542,1114097888,IC_kwDOAMm_X85CZ8jg,3698640,2022-05-01T01:47:27Z,2022-05-01T01:47:27Z,CONTRIBUTOR,"sorry I know I said I'd fix this but I'm having a very hard time figuring out what is wrong and how to build the docs. I had to set up a docker image to build them because I couldn't get the ipython directive to use the right conda env on my laptop, and now I'm getting a version error when pandas encounters the default build number on my fork. I'm a bit embarrassed that I can't figure this out, but... I think I might need a hand getting this across the finish line 😕 ``` .... #13 1597.6 reading sources... [ 97%] getting-started-guide/faq #13 1600.8 reading sources... [ 97%] getting-started-guide/index #13 1600.8 reading sources... [ 97%] getting-started-guide/installing #13 1601.0 reading sources... [ 97%] getting-started-guide/quick-overview #13 1609.3 WARNING: #13 1609.3 >>>------------------------------------------------------------------------- #13 1609.3 Exception in /xarray/doc/getting-started-guide/quick-overview.rst at block ending on line 169 #13 1609.3 Specify :okexcept: as an option in the ipython:: block to suppress this message #13 1609.3 --------------------------------------------------------------------------- #13 1609.3 ImportError Traceback (most recent call last) #13 1609.3 Input In [40], in () #13 1609.3 ----> 1 series.to_xarray() #13 1609.3 #13 1609.3 File /srv/conda/envs/xarray-doc/lib/python3.9/site-packages/pandas/core/generic.py:3173, in NDFrame.to_xarray(self) #13 1609.3 3096 @final #13 1609.3 3097 def to_xarray(self): #13 1609.3 3098 """""" #13 1609.3 3099 Return an xarray object from the pandas object. #13 1609.3 3100 #13 1609.3 (...) #13 1609.3 3171 speed (date, animal) int64 350 18 361 15 #13 1609.3 3172 """""" #13 1609.3 -> 3173 xarray = import_optional_dependency(""xarray"") #13 1609.3 3175 if self.ndim == 1: #13 1609.3 3176 return xarray.DataArray.from_series(self) #13 1609.3 #13 1609.3 File /srv/conda/envs/xarray-doc/lib/python3.9/site-packages/pandas/compat/_optional.py:164, in import_optional_dependency(name, extra, errors, min_version) #13 1609.3 162 return None #13 1609.3 163 elif errors == ""raise"": #13 1609.3 --> 164 raise ImportError(msg) #13 1609.3 166 return module #13 1609.3 #13 1609.3 ImportError: Pandas requires version '0.15.1' or newer of 'xarray' (version '0.1.dev1+gfdf7303' currently installed). #13 1609.3 #13 1609.3 <<<------------------------------------------------------------------------- #13 1609.3 #13 1609.3 Exception occurred: #13 1609.3 File ""/srv/conda/envs/xarray-doc/lib/python3.9/site-packages/IPython/sphinxext/ipython_directive.py"", line 584, in process_input #13 1609.3 raise RuntimeError('Non Expected exception in `{}` line {}'.format(filename, lineno)) #13 1609.3 RuntimeError: Non Expected exception in `/xarray/doc/getting-started-guide/quick-overview.rst` line 169 #13 1609.3 The full traceback has been saved in /tmp/sphinx-err-a01y480j.log, if you want to report the issue to the developers. #13 1609.3 Please also report this if it was a user error, so that a better error message can be provided next time. #13 1609.3 A bug report can be filed in the tracker at . Thanks! #13 1611.4 make: *** [Makefile:58: html] Error 2 #13 1611.6 ERROR conda.cli.main_run:execute(41): `conda run /bin/bash -c make html` failed. (See above for error) #13 ERROR: executor failed running [conda run --no-capture-output -n xarray-doc /bin/bash -c make html]: exit code: 2 ------ > [10/10] RUN make html: ------ executor failed running [conda run --no-capture-output -n xarray-doc /bin/bash -c make html]: exit code: 2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/pull/6542#issuecomment-1113712207,https://api.github.com/repos/pydata/xarray/issues/6542,1113712207,IC_kwDOAMm_X85CYeZP,3698640,2022-04-29T20:44:43Z,2022-04-29T20:44:43Z,CONTRIBUTOR,hmmm seems I've messed something up in the docs build. apologize for the churn - will fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1221393104 https://github.com/pydata/xarray/issues/6456#issuecomment-1098574761,https://api.github.com/repos/pydata/xarray/issues/6456,1098574761,IC_kwDOAMm_X85Beuup,3698640,2022-04-13T23:34:16Z,2022-04-13T23:34:48Z,CONTRIBUTOR,"> In the example it's saving every iteration, but in my actual code it's much less frequent when I said ""you're overwriting the file every iteration"" I meant to put the emphasis on _overwiting_. by using `mode='w'` instead of `mode='a'` you're telling zarr to delete the file if it exists and the re-create it every time `to_zarr` is executed. See the docs on [`xr.Dataset.to_zarr`](https://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_zarr.html): > **mode** (`{""w"", ""w-"", ""a"", ""r+"", None}`, *optional*) – Persistence mode: “w” means create (overwrite if exists); “w-” means create (fail if exists); “a” means override existing variables (create if does not exist); “r+” means modify existing array values only (raise an error if any metadata or shapes would change). The default mode is “a” if `append_dim` is set. Otherwise, it is “r+” if `region` is set and `w-` otherwise. This interpretation of mode is consistent across all of python - see the docs for [python builtins: open](https://docs.python.org/3/library/functions.html#open) So I think changing your writes to `ds3.to_zarr('zarr_bug.zarr', mode='a')` as Max suggested will get you a good part of the way there :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1197117301 https://github.com/pydata/xarray/pull/6467#issuecomment-1096953860,https://api.github.com/repos/pydata/xarray/issues/6467,1096953860,IC_kwDOAMm_X85BYjAE,3698640,2022-04-12T16:39:53Z,2022-04-12T16:39:53Z,CONTRIBUTOR,"oof - debugging CI build process is no fun. good luck, and thanks @max-sixty !!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1199127752 https://github.com/pydata/xarray/issues/6456#issuecomment-1094411214,https://api.github.com/repos/pydata/xarray/issues/6456,1094411214,IC_kwDOAMm_X85BO2PO,3698640,2022-04-10T23:40:49Z,2022-04-10T23:40:49Z,CONTRIBUTOR,"@tbloch1 following up on Max's suggestion - it looks like you might be overwriting the file with every iteration. See the docs on [ds.to_zarr](https://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_zarr.html) - `mode='w'` will overwrite the file while `mode='a'` will append. That said, you still would need your indices to not overlap. How are you distinguishing between the files? is each one a different point in time? To me, this doesn't seem likely to be a bug, but is more of a usage question. Have you tried asking on stackoverflow with the xarray tag? ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1197117301 https://github.com/pydata/xarray/pull/6467#issuecomment-1094409424,https://api.github.com/repos/pydata/xarray/issues/6467,1094409424,IC_kwDOAMm_X85BO1zQ,3698640,2022-04-10T23:29:38Z,2022-04-10T23:29:38Z,CONTRIBUTOR,hmm. readthedocs failed because of concurrency limits (my bad) but seems to have failed to automatically retry. can someone give it a nudge?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1199127752 https://github.com/pydata/xarray/pull/6467#issuecomment-1094342045,https://api.github.com/repos/pydata/xarray/issues/6467,1094342045,IC_kwDOAMm_X85BOlWd,3698640,2022-04-10T18:26:32Z,2022-04-10T18:26:32Z,CONTRIBUTOR,"thanks @max-sixty! I was mostly trying to be overly-cautious because the ValueError was so clearly raised intentionally, and I can't figure out why it was necessary. But maybe it was just left over from a time when providing both arguments really would have caused a problem, and it's no longer the case.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1199127752 https://github.com/pydata/xarray/issues/6036#issuecomment-1005162696,https://api.github.com/repos/pydata/xarray/issues/6036,1005162696,IC_kwDOAMm_X8476ZDI,3698640,2022-01-04T20:53:36Z,2022-01-04T20:54:13Z,CONTRIBUTOR,"This isn't a fix for the overhead required to manage an arbitrarily large graph, but note that creating chunks this small (size 1 in this case) is explicitly not recommended. See the dask docs on [Array Best Practices: Select a good chunk size](https://docs.dask.org/en/stable/array-best-practices.html#select-a-good-chunk-size) - they recommend chunks *no smaller than 100 MB*. Your chunks are 8 bytes. This creates 1 billion tasks, which does result in an enormous overhead - there's no way around this. Note that storing this on disk would not help - the problem results from the fact that 1 billion tasks will almost certainly overwhelm any dask scheduler. The general [dask best practices](https://docs.dask.org/en/stable/array-best-practices.html) guide recommends keeping the number of tasks below 1 million if possible. Also, I don't think that the issue here is in specifying the universe of the tasks that need to be created, but rather in creating and managing the python task objects themselves. So pre-computing or storing them wouldn't help. For me, changing to (1000, 1000, 100) chunks (~750MB for a float64 array) reduces the time to a couple ms: ```python In [16]: %%timeit ...: ...: chunks = (1000, 1000, 100) ...: ds = xr.Dataset(data_vars={ ...: ""foo"": (('x', 'y', 'z'), dask.array.empty((1000, 1000, 1000), chunks=(1000, 1000, 1000)))}) ...: ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}, mode='w') ...: ds_loaded = xr.open_zarr(group='ds.zarr', store='data') ...: ...: 6.36 ms ± 111 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` With this chunking scheme, you could store and work with much, much more data. In fact, scaling the size of your example by 3 orders of magnitude only increases the runtime by ~5x: ```python In [18]: %%timeit ...: ...: chunks = (1000, 1000, 100, 1) ...: ds = xr.Dataset(data_vars={ ...: ""foo"": (('w', 'x', 'y', 'z'), dask.array.empty((1000, 1000, 1000, 1000), chunks=(1000, 1000, 1000, 1)))}) ...: ds.to_zarr(store='data', group='ds.zarr', compute=False, encoding={'foo': {'chunks': chunks}}, mode='w') ...: ds_loaded = xr.open_zarr(group='ds.zarr', store='data') ...: ...: 36.9 ms ± 2.23 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) ``` So if re-writing your arrays with larger chunks is an option I think this could get around the problem you're seeing?","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1068225524 https://github.com/pydata/xarray/issues/5994#issuecomment-1003460778,https://api.github.com/repos/pydata/xarray/issues/5994,1003460778,IC_kwDOAMm_X847z5iq,3698640,2021-12-31T22:36:04Z,2021-12-31T22:41:51Z,CONTRIBUTOR,"This looks like it could be a good improvement! Just flagging that in [`func_interpolate_na` on main](https://github.com/pydata/xarray/blob/main/xarray/core/missing.py#L379-L396) I see a different condition: ```python # fast track for no-nans and all-nans cases n_nans = nans.sum() if n_nans == 0 or n_nans == len(y): return y ``` Do you see the `n_nans < 0` condition somewhere or is this just a typo? That condition also wouldn't make much sense. Your point about using the shortcut for `n_nans == len(y) - 1` seems right to me, and definitely doesn't seem worthy of an error when an all-NaN slice is silently skipped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1055867960 https://github.com/pydata/xarray/issues/6124#issuecomment-1002777049,https://api.github.com/repos/pydata/xarray/issues/6124,1002777049,IC_kwDOAMm_X847xSnZ,3698640,2021-12-29T21:09:12Z,2021-12-29T21:09:12Z,CONTRIBUTOR,"I realize this may be a larger discussion, but the implementation is so easy I went ahead and filed a PR that issues a PendingDeprecationWarning in `Dataset.__bool__`.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,1090229430 https://github.com/pydata/xarray/issues/6124#issuecomment-1002673070,https://api.github.com/repos/pydata/xarray/issues/6124,1002673070,IC_kwDOAMm_X847w5Ou,3698640,2021-12-29T16:22:10Z,2021-12-29T16:22:10Z,CONTRIBUTOR,"> If there's a synthesis of keeping the truthiness while reducing the chance of these mistakes, that would be very welcome. @max-sixty im not sure what this would look like. Do you mean a warning or are you hinting that the bar that would need to be met is a silver bullet that preserves bool(ds) but somehow isn’t confusing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1090229430 https://github.com/pydata/xarray/issues/6124#issuecomment-1002671461,https://api.github.com/repos/pydata/xarray/issues/6124,1002671461,IC_kwDOAMm_X847w41l,3698640,2021-12-29T16:18:35Z,2021-12-29T16:18:35Z,CONTRIBUTOR,"Yeah… I do understand how it’s currently working and why, and the behavior is certainly intuitive to those who appreciate the mapping inheritance. That said, I feel I have to make a last stand argument because this trips people up quite often (on my team and elsewhere). I haven’t yet come across an example of anyone using this correctly, but I see users misusing it all the time. The examples and behavior you’re showing @Illviljan seem to me like more the natural result of an implementation detail than a critical principle of the dataset design. While it’s obvious why `bool(ds)` resolves to True or False in the current implementation, I still think it’s not used and is out of step with how most users think of a dataset. The fact that nothing in the package requires a change in order to pass tests (the broken tests and the pathological case aside) reaffirms for me that this is not a useful feature. I don’t know much about the mapping protocol or how closely it must be followed. Is the idea here that packages building on xarray (or interoperability features in e.g. numpy or dask) depend on a strict adherence to the full spec? ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1090229430 https://github.com/pydata/xarray/issues/3951#issuecomment-610615488,https://api.github.com/repos/pydata/xarray/issues/3951,610615488,MDEyOklzc3VlQ29tbWVudDYxMDYxNTQ4OA==,3698640,2020-04-07T20:55:11Z,2020-04-07T20:55:11Z,CONTRIBUTOR,"Here's a test script I'm using for this: ```bash echo '$ conda create -n py37xr14 -c conda-forge --yes python=3.7 xarray=0.14.1' conda create -n py37xr14 -c conda-forge --yes python=3.7 xarray=0.14.1 > /dev/null echo '$ conda create -n py37xr15 -c conda-forge --yes python=3.7 xarray=0.15.1' conda create -n py37xr15 -c conda-forge --yes python=3.7 xarray=0.15.1 > /dev/null echo '$ conda run -n py37xr14 python test.py' conda run -n py37xr14 python test.py echo echo '$ conda run -n py37xr15 python test.py' conda run -n py37xr15 python test.py echo echo '$ conda list -n py37xr14' conda list -n py37xr14 echo echo '$ conda list -n py37xr15' conda list -n py37xr15 conda env remove -n py37xr14 > /dev/null 2>&1 conda env remove -n py37xr15 > /dev/null 2>&1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,596115014 https://github.com/pydata/xarray/issues/3951#issuecomment-610612332,https://api.github.com/repos/pydata/xarray/issues/3951,610612332,MDEyOklzc3VlQ29tbWVudDYxMDYxMjMzMg==,3698640,2020-04-07T20:47:51Z,2020-04-07T20:47:51Z,CONTRIBUTOR,yeah I use this pattern all the time - df.stack().to_xarray() seems to now fail unless your columns were sorted alphabetically. not sure yet where this is happening but it does result in some pernicious bad data errors that can be hard to debug if you catch them at all.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,596115014 https://github.com/pydata/xarray/issues/1075#issuecomment-373541528,https://api.github.com/repos/pydata/xarray/issues/1075,373541528,MDEyOklzc3VlQ29tbWVudDM3MzU0MTUyOA==,3698640,2018-03-15T22:21:51Z,2018-03-16T01:33:24Z,CONTRIBUTOR,"xarray==0.10.2 netCDF4==1.3.1 Just tried it again and didn't have any issues: ```python patt = ( 'http://nasanex.s3.amazonaws.com/NEX-GDDP/BCSD/{scen}/day/atmos/{var}/' + 'r1i1p1/v1.0/{var}_day_BCSD_{scen}_r1i1p1_{model}_{year}.nc') def open_url_dataset(url): fname = os.path.splitext(os.path.basename(url))[0] res = requests.get(url) content = io.BytesIO(res.content) nc4_ds = netCDF4.Dataset(fname, memory=res.content) store = xr.backends.NetCDF4DataStore(nc4_ds) ds = xr.open_dataset(store) return ds ds = open_url_dataset(url=patt.format( model='GFDL-ESM2G', scen='historical', var='tasmax', year=1988)) ds ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-357125148,https://api.github.com/repos/pydata/xarray/issues/1075,357125148,MDEyOklzc3VlQ29tbWVudDM1NzEyNTE0OA==,3698640,2018-01-12T02:27:27Z,2018-01-12T02:27:27Z,CONTRIBUTOR,"yes! Thanks @jhamman and @shoyer. I hadn't tried it yet, but just did. worked great! ```python In [1]: import xarray as xr ...: import requests ...: import netCDF4 ...: ...: %matplotlib inline In [2]: res = requests.get( ...: 'http://nasanex.s3.amazonaws.com/NEX-GDDP/BCSD/rcp45/day/atmos/tasmin/' + ...: 'r1i1p1/v1.0/tasmin_day_BCSD_rcp45_r1i1p1_CESM1-BGC_2073.nc') In [3]: res.status_code Out [3]: 200 In [4]: res.headers['content-type'] Out [4]: 'application/x-netcdf' In [5]: nc4_ds = netCDF4.Dataset('tasmin_day_BCSD_rcp45_r1i1p1_CESM1-BGC_2073', memory=res.content) In [6]: store = xr.backends.NetCDF4DataStore(nc4_ds) In [7]: ds = xr.open_dataset(store) In [8]: ds.tasmin.isel(time=0).plot() /global/home/users/mdelgado/git/public/xarray/xarray/plot/utils.py:51: FutureWarning: 'pandas.tseries.converter.register' has been moved and renamed to 'pandas.plotting.register_matplotlib_converters'. converter.register() Out [8]: ``` ![output_7_2](https://user-images.githubusercontent.com/3698640/34856943-f82619f4-f6fc-11e7-831d-f5d4032a338a.png) ```python In [9]: ds Out [9]: Dimensions: (lat: 720, lon: 1440, time: 365) Coordinates: * time (time) datetime64[ns] 2073-01-01T12:00:00 2073-01-02T12:00:00 ... * lat (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ... * lon (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ... Data variables: tasmin (time, lat, lon) float64 ... Attributes: parent_experiment: historical parent_experiment_id: historical parent_experiment_rip: r1i1p1 Conventions: CF-1.4 institution: NASA Earth Exchange, NASA Ames Research C... institute_id: NASA-Ames realm: atmos modeling_realm: atmos version: 1.0 downscalingModel: BCSD experiment_id: rcp45 frequency: day realization: 1 initialization_method: 1 physics_version: 1 tracking_id: 1865ff49-b20c-4268-852a-a9503efec72c driving_data_tracking_ids: N/A driving_model_ensemble_member: r1i1p1 driving_experiment_name: historical driving_experiment: historical model_id: BCSD references: BCSD method: Thrasher et al., 2012, Hydro... DOI: http://dx.doi.org/10.7292/W0MW2F2G experiment: RCP4.5 title: CESM1-BGC global downscaled NEX CMIP5 Cli... contact: Dr. Rama Nemani: rama.nemani@nasa.gov, Dr... disclaimer: This data is considered provisional and s... resolution_id: 0.25 degree project_id: NEXGDDP table_id: Table day (12 November 2010) source: BCSD 2014 creation_date: 2015-01-07T19:18:31Z forcing: N/A product: output ``` ","{""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/pull/1802#issuecomment-354469859,https://api.github.com/repos/pydata/xarray/issues/1802,354469859,MDEyOklzc3VlQ29tbWVudDM1NDQ2OTg1OQ==,3698640,2017-12-29T16:45:37Z,2017-12-29T16:45:37Z,CONTRIBUTOR,Ok this is good to go if you all do want to enable _FillValue for variable-length unicode strings with a netCDF4 backend. Seems like there's a lot of prior work/thinking in this space though so no worries if you want to wait.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,285006452 https://github.com/pydata/xarray/pull/1802#issuecomment-354379246,https://api.github.com/repos/pydata/xarray/issues/1802,354379246,MDEyOklzc3VlQ29tbWVudDM1NDM3OTI0Ng==,3698640,2017-12-29T00:22:57Z,2017-12-29T00:22:57Z,CONTRIBUTOR,lol. no I'm just walking around in your footsteps @shoyer. I've just enabled the tests you presumably wrote for #1647 & #1648. Curious why variable-length unicode strings with _FillHoles using netCDF4 doesn't currently work in master?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,285006452 https://github.com/pydata/xarray/pull/1802#issuecomment-354378158,https://api.github.com/repos/pydata/xarray/issues/1802,354378158,MDEyOklzc3VlQ29tbWVudDM1NDM3ODE1OA==,3698640,2017-12-29T00:11:53Z,2017-12-29T00:11:53Z,CONTRIBUTOR,"hmm. Seems I'm touching on a much larger issue here: Unidata/netcdf4-python#730 The round-trip works for me using a netcdf4 engine once this fix is implemented in conventions.py. There are tests that are ready to demonstrate this in test_backends.py:836-843, but running these tests (by removing the `pytest.raises` lines) applies to both netCDF4 and h5netcdf backends. Should these use cases be split up?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,285006452 https://github.com/pydata/xarray/issues/1781#issuecomment-354370636,https://api.github.com/repos/pydata/xarray/issues/1781,354370636,MDEyOklzc3VlQ29tbWVudDM1NDM3MDYzNg==,3698640,2017-12-28T22:55:28Z,2017-12-28T22:55:28Z,CONTRIBUTOR,"I've got a reproducible example of this (sorry for the length): ```python In [1]: import xarray as xr ...: import numpy as np ...: import pandas as pd ...: import netCDF4 ``` ```python In [2]: xr.show_versions() ``` INSTALLED VERSIONS ------------------ commit: None python: 3.5.4.final.0 python-bits: 64 OS: Darwin OS-release: 17.3.0 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 xarray: 0.10.0 pandas: 0.21.0 numpy: 1.13.3 scipy: 1.0.0 netCDF4: 1.3.1 h5netcdf: None Nio: None bottleneck: 1.2.1 cyordereddict: None dask: 0.16.0 matplotlib: 2.1.0 cartopy: None seaborn: 0.8.1 setuptools: 36.5.0.post20170921 pip: 9.0.1 conda: None pytest: 3.3.1 IPython: 6.2.1 sphinx: None ```python In [3]: nds = netCDF4.Dataset('test.nc', 'w', format='NETCDF4') ``` ```python In [4]: nds.createDimension('X', 3) ...: nds.createVariable('X', str, ('X', ), fill_value='invalid') ...: nds.createVariable('xname', str, ('X', ), fill_value='invalid') ...: nds.createVariable('var1', np.float32, ('X', ), fill_value=np.nan) ...: ...: nds.variables['X'][:] = np.array(['first', 'second', 'third']) ...: nds.variables['xname'][:] = np.array(['the_first', 'invalid', 'the_third']) ...: nds.variables['var1'][:] = np.random.random((3, )) ``` ```python In [5]: nds.variables['X'] ``` vlen X(X) _FillValue: invalid vlen data type: unlimited dimensions: current shape = (3,) ```python In [6]: nds.variables['X'][:] ``` array(['first', 'second', 'third'], dtype=object) ```python In [7]: nds.variables['xname'] ``` vlen xname(X) _FillValue: invalid vlen data type: unlimited dimensions: current shape = (3,) ```python In [8]: nds.variables['xname'][:] ``` array(['the_first', 'invalid', 'the_third'], dtype=object) ```python In [9]: nds.variables['var1'] ``` float32 var1(X) _FillValue: nan unlimited dimensions: current shape = (3,) filling on ```python In [10]: nds.variables['var1'][:] ``` array([ 0.58315825, 0.82443351, 0.39345944], dtype=float32) ```python In [11]: nds.close() ``` ```python In [12]: ds = xr.open_dataset('test.nc') ``` --------------------------------------------------------------------------- UnboundLocalError Traceback (most recent call last) in () ----> 1 ds = xr.open_dataset('test.nc') ~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables) 303 lock = _default_lock(filename_or_obj, engine) 304 with close_on_error(store): --> 305 return maybe_decode_store(store, lock) 306 else: 307 if engine is not None and engine != 'scipy': ~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/backends/api.py in maybe_decode_store(store, lock) 223 store, mask_and_scale=mask_and_scale, decode_times=decode_times, 224 concat_characters=concat_characters, decode_coords=decode_coords, --> 225 drop_variables=drop_variables) 226 227 _protect_dataset_variables_inplace(ds, cache) ~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables) 1153 vars, attrs, coord_names = decode_cf_variables( 1154 vars, attrs, concat_characters, mask_and_scale, decode_times, -> 1155 decode_coords, drop_variables=drop_variables) 1156 ds = Dataset(vars, attrs=attrs) 1157 ds = ds.set_coords(coord_names.union(extra_coords).intersection(vars)) ~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables) 1086 k, v, concat_characters=concat_characters, 1087 mask_and_scale=mask_and_scale, decode_times=decode_times, -> 1088 stack_char_dim=stack_char_dim) 1089 if decode_coords: 1090 var_attrs = new_vars[k].attrs ~/miniconda2/envs/spatial3/lib/python3.5/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim) 998 if (has_fill or scale_factor is not None or add_offset is not None): 999 if has_fill and np.array(fill_value).dtype.kind in ['U', 'S', 'O']: -> 1000 if string_encoding is not None: 1001 raise NotImplementedError( 1002 'variable %r has a _FillValue specified, but ' UnboundLocalError: local variable 'string_encoding' referenced before assignment This seems to be produced by the fact that `string_encoding` is not defined for unicode-encoded variable-length string types. Could the problem be the lack of `'O'` and `'U'` in conventions.py line [952](https://github.com/pydata/xarray/blob/6eac8574f85623b18b59ba1ac9cf95e09c87980b/xarray/conventions.py#L952)? I'll try my hand at a PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,282061228 https://github.com/pydata/xarray/issues/1075#issuecomment-347705483,https://api.github.com/repos/pydata/xarray/issues/1075,347705483,MDEyOklzc3VlQ29tbWVudDM0NzcwNTQ4Mw==,3698640,2017-11-28T23:58:41Z,2017-11-28T23:58:41Z,CONTRIBUTOR,"Thanks @shoyer. So you can download the entire object into memory and then create a file image and read that? While not a full fix, it's definitely an improvement over download-to-disk-then-read workflow!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1472#issuecomment-314154076,https://api.github.com/repos/pydata/xarray/issues/1472,314154076,MDEyOklzc3VlQ29tbWVudDMxNDE1NDA3Ng==,3698640,2017-07-10T16:08:30Z,2017-07-10T16:08:30Z,CONTRIBUTOR,"I wasn't misled by the docs, just by my intuition. But now that you've made the distinction between `'2012'` and `'2012-12-31'` this totally makes sense: ```python In [24]: arr.sel(time='2012') Out[24]: array([[ 0.244146, 0.702819, 0.06614 , 0.211758]]) Coordinates: * age (age) object 'age0' 'age1' 'age2' 'age3' * time (time) datetime64[ns] 2012-12-31 In [25]: arr.sel(time='2012-12-31') Out[25]: array([ 0.244146, 0.702819, 0.06614 , 0.211758]) Coordinates: * age (age) object 'age0' 'age1' 'age2' 'age3' time datetime64[ns] 2012-12-31 In [26]: arr.sel(time=np.datetime64('2012-12-31')) Out[26]: array([ 0.244146, 0.702819, 0.06614 , 0.211758]) Coordinates: * age (age) object 'age0' 'age1' 'age2' 'age3' time datetime64[ns] 2012-12-31 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,241389297 https://github.com/pydata/xarray/issues/1472#issuecomment-314136206,https://api.github.com/repos/pydata/xarray/issues/1472,314136206,MDEyOklzc3VlQ29tbWVudDMxNDEzNjIwNg==,3698640,2017-07-10T15:10:50Z,2017-07-10T15:10:50Z,CONTRIBUTOR,thanks for the schooling. learned something new!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,241389297 https://github.com/pydata/xarray/issues/1075#issuecomment-258025809,https://api.github.com/repos/pydata/xarray/issues/1075,258025809,MDEyOklzc3VlQ29tbWVudDI1ODAyNTgwOQ==,3698640,2016-11-02T23:03:34Z,2016-11-02T23:03:34Z,CONTRIBUTOR,"Got it. :( Thanks! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/798#issuecomment-257062571,https://api.github.com/repos/pydata/xarray/issues/798,257062571,MDEyOklzc3VlQ29tbWVudDI1NzA2MjU3MQ==,3698640,2016-10-29T01:26:47Z,2016-10-29T01:26:47Z,CONTRIBUTOR,"Could this extend the OpeNDAP interface? That would solve the metadata problem and would provide quick access to the distributed workers. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,142498006