issues: 474582412
This data as json
id | node_id | number | title | user | state | locked | assignee | milestone | comments | created_at | updated_at | closed_at | author_association | active_lock_reason | draft | pull_request | body | reactions | performed_via_github_app | state_reason | repo | type |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
474582412 | MDU6SXNzdWU0NzQ1ODI0MTI= | 3170 | Dataset.to_zarr() with mode='a' does not work with groups | 18643609 | closed | 0 | 0 | 2019-07-30T13:22:58Z | 2020-03-02T12:19:16Z | 2020-03-02T12:19:16Z | NONE | MCVE Code Sample```python import xarray as xr import numpy as np from s3fs import S3FileSystem, S3Map s3 = S3FileSystem() bucket_name = 'your-bucket-name' s3_path = bucket_name + 'some_path.zarr' store = S3Map(s3_path, s3=s3) for i in range(6): if i%2 == 0: group = 'Group1' else: group = 'Group2' lead_time = i//2 var1 = np.random.rand(1) ds = xr.Dataset({'var1': (['lead_time'], var1)}, coords={'lead_time': [lead_time]}) ds.to_zarr(store=store, mode='a', append_dim='lead_time', group=group) ``` OutputThis code returns the following error:
```
Traceback (most recent call last):
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/core/dataset.py", line 1019, in _construct_dataarray
variable = self._variables[name]
KeyError: 'var1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/vincent/Documents/Greenlytics/SiteForecast/debugging-script.py", line 201, in <module>
ds.to_zarr(store=store, mode='a', append_dim='lead_time', group=group)
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/core/dataset.py", line 1433, in to_zarr
consolidated=consolidated, append_dim=append_dim)
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/backends/api.py", line 1101, in to_zarr
dump_to_store(dataset, zstore, writer, encoding=encoding)
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/backends/api.py", line 929, in dump_to_store
unlimited_dims=unlimited_dims)
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/backends/zarr.py", line 358, in store
variables_with_encoding[vn].encoding = ds[vn].encoding
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/core/dataset.py", line 1103, in __getitem__
return self._construct_dataarray(key)
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/core/dataset.py", line 1022, in _construct_dataarray
self._variables, name, self._level_coords, self.dims)
File "/home/vincent/anaconda3/envs/hanover_backend/lib/python3.7/site-packages/xarray/core/dataset.py", line 91, in _get_virtual_variable
ref_var = variables[ref_name]
KeyError: 'var1'
```
The KeyError can happen on a variable name as well as on a dimension name, it depends on the runs. Problem DescriptionI am trying to use the append mode introduced in the PR #2706 on zarr groups. This raises a KeyError as you can see in the trace above. Is it a bug or a feature that is not supported (yet)? Output of
|
{ "url": "https://api.github.com/repos/pydata/xarray/issues/3170/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
completed | 13221727 | issue |