issue_comments

Table actions

11 rows where author_association = "CONTRIBUTOR" and issue = 573577844 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • Opening from zarr.ZipStore fails to read (store???) unicode characters · 11 ✖

author_association 1

  • CONTRIBUTOR · 11 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
604181264 https://github.com/pydata/xarray/issues/3815#issuecomment-604181264 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwNDE4MTI2NA== hmaarrfk 90008 2020-03-26T01:49:45Z 2020-03-26T01:49:45Z CONTRIBUTOR

And actually, zarr provides a data argument in create_dataset that actually encounters the same bug python import zarr import numpy as np name = 'hello' data = np.array('world', dtype='<U5') store = zarr.ZipStore('test_store.zip', mode='w') root = zarr.open(store , mode='w') zarr_array = root.create_dataset(name, data=data, shape=data.shape, dtype=data.dtype) zarr_array[...]

I guess i can open upstream in zarr, but I think for catching the 0 sized array case, it is probably best to use the data argument.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
604180511 https://github.com/pydata/xarray/issues/3815#issuecomment-604180511 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwNDE4MDUxMQ== hmaarrfk 90008 2020-03-26T01:46:52Z 2020-03-26T01:46:52Z CONTRIBUTOR

I think the reason is that for zero sized arrays, you technically aren't allowed to write data to them.

https://github.com/pydata/xarray/blob/6378a711d50ba7f1ba9b2a451d4d1f5e1fb37353/xarray/backends/zarr.py#L449

This means that when you create the 0 sized array, you can't actually change the value.

Here is a reproducer without xarray python import zarr import numpy as np name = 'hello' data = np.array('world', dtype='<U5') store = zarr.ZipStore('test_store.zip', mode='w') root = zarr.open(store , mode='w') zarr_array = root.create_dataset(name, shape=data.shape, dtype=data.dtype) root[name][...] = data zarr_array[...]

Though the code path follows what xarray does in the backend.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
604009141 https://github.com/pydata/xarray/issues/3815#issuecomment-604009141 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwNDAwOTE0MQ== hmaarrfk 90008 2020-03-25T18:26:58Z 2020-03-25T18:26:58Z CONTRIBUTOR

@jakirkham not sure if you have any thoughts on why the code above is bugging out.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
604008658 https://github.com/pydata/xarray/issues/3815#issuecomment-604008658 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwNDAwODY1OA== hmaarrfk 90008 2020-03-25T18:26:10Z 2020-03-25T18:26:10Z CONTRIBUTOR

Honestly, i've found that __ coordinates are treated as less strict coordinates, so I've been using those.

Keeping things like: * The version of different software that was used.

Seems more like an attribute, but really, it is all data, so :/.

Regarding the originally issue, it seems that you are right in the sense that a 0 dimension string might be buggy in zarr itself.

I guess we (when we have time) will have to dig down to find a MVC example that reproduces the issue without xarray to submit to Zarr.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603921577 https://github.com/pydata/xarray/issues/3815#issuecomment-603921577 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzkyMTU3Nw== hmaarrfk 90008 2020-03-25T15:54:37Z 2020-03-25T15:54:37Z CONTRIBUTOR

Hmm, interesting!

I've avoided attrs since they often get "lost" in computation, and don't get dragged along as rigorously as coordinates.

I do have some real coordinates that are stored as strings.

Thanks for the quickfeedback.

Here is the reproducing code without using context managers (which auto clsoe things you know)

```python import xarray as xr import zarr x = xr.Dataset() x['hello'] = 'world' x with zarr.ZipStore('test_store.zip', mode='w') as store: x.to_zarr(store)

read_store = zarr.ZipStore('test_store.zip', mode='r')
x_read = xr.open_zarr(read_store).compute()

The error will happen before this line is executed

read_store.close()

```

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603608958 https://github.com/pydata/xarray/issues/3815#issuecomment-603608958 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzYwODk1OA== hmaarrfk 90008 2020-03-25T02:45:12Z 2020-03-25T02:45:12Z CONTRIBUTOR

I will have to try the debugging things you mentionned some later time :/

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603608822 https://github.com/pydata/xarray/issues/3815#issuecomment-603608822 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzYwODgyMg== hmaarrfk 90008 2020-03-25T02:44:40Z 2020-03-25T02:44:40Z CONTRIBUTOR

Not sure if the builds in https://github.com/pydata/xarray/pull/3888 help reproduce things or not?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603601762 https://github.com/pydata/xarray/issues/3815#issuecomment-603601762 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzYwMTc2Mg== hmaarrfk 90008 2020-03-25T02:16:29Z 2020-03-25T02:16:29Z CONTRIBUTOR

hmm i didn't realize this. I"m running from conda-forge + linux.

Let me try on your CIs.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603556048 https://github.com/pydata/xarray/issues/3815#issuecomment-603556048 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzU1NjA0OA== hmaarrfk 90008 2020-03-24T23:24:53Z 2020-03-24T23:24:53Z CONTRIBUTOR

See the zipstore example in my first comment

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603555953 https://github.com/pydata/xarray/issues/3815#issuecomment-603555953 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzU1NTk1Mw== hmaarrfk 90008 2020-03-24T23:24:34Z 2020-03-24T23:24:34Z CONTRIBUTOR

I thought I provided it, but in either case, here is my traceback

```python In [3]: import xarray as xr ...: import zarr ...: x = xr.Dataset() ...: x['hello'] = 'world' ...: x ...: with zarr.ZipStore('test_store.zip', mode='w') as store: ...: x.to_zarr(store) ...: with zarr.ZipStore('test_store.zip', mode='r') as store: ...: x_read = xr.open_zarr(store).compute() ...: --------------------------------------------------------------------------- BadZipFile Traceback (most recent call last) <ipython-input-3-5ad5a0456766> in <module> 7 x.to_zarr(store) 8 with zarr.ZipStore('test_store.zip', mode='r') as store: ----> 9 x_read = xr.open_zarr(store).compute() 10 ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/xarray/core/dataset.py in compute(self, **kwargs) 805 """ 806 new = self.copy(deep=False) --> 807 return new.load(**kwargs) 808 809 def _persist_inplace(self, **kwargs) -> "Dataset": ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/xarray/core/dataset.py in load(self, **kwargs) 657 for k, v in self.variables.items(): 658 if k not in lazy_data: --> 659 v.load() 660 661 return self ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/xarray/core/variable.py in load(self, **kwargs) 373 self._data = as_compatible_data(self._data.compute(**kwargs)) 374 elif not hasattr(self._data, "__array_function__"): --> 375 self._data = np.asarray(self._data) 376 return self 377 ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87 ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/xarray/core/indexing.py in __array__(self, dtype) 555 def __array__(self, dtype=None): 556 array = as_indexable(self.array) --> 557 return np.asarray(array[self.key], dtype=None) 558 559 def transpose(self, order): ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/xarray/backends/zarr.py in __getitem__(self, key) 47 array = self.get_array() 48 if isinstance(key, indexing.BasicIndexer): ---> 49 return array[key.tuple] 50 elif isinstance(key, indexing.VectorizedIndexer): 51 return array.vindex[ ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/zarr/core.py in __getitem__(self, selection) 570 571 fields, selection = pop_fields(selection) --> 572 return self.get_basic_selection(selection, fields=fields) 573 574 def get_basic_selection(self, selection=Ellipsis, out=None, fields=None): ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/zarr/core.py in get_basic_selection(self, selection, out, fields) 693 if self._shape == (): 694 return self._get_basic_selection_zd(selection=selection, out=out, --> 695 fields=fields) 696 else: 697 return self._get_basic_selection_nd(selection=selection, out=out, ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/zarr/core.py in _get_basic_selection_zd(self, selection, out, fields) 709 # obtain encoded data for chunk 710 ckey = self._chunk_key((0,)) --> 711 cdata = self.chunk_store[ckey] 712 713 except KeyError: ~/miniconda3/envs/mcam_dev/lib/python3.7/site-packages/zarr/storage.py in __getitem__(self, key) 1249 with self.mutex: 1250 with self.zf.open(key) as f: # will raise KeyError -> 1251 return f.read() 1252 1253 def __setitem__(self, key, value): ~/miniconda3/envs/mcam_dev/lib/python3.7/zipfile.py in read(self, n) 914 self._offset = 0 915 while not self._eof: --> 916 buf += self._read1(self.MAX_N) 917 return buf 918 ~/miniconda3/envs/mcam_dev/lib/python3.7/zipfile.py in _read1(self, n) 1018 if self._left <= 0: 1019 self._eof = True -> 1020 self._update_crc(data) 1021 return data 1022 ~/miniconda3/envs/mcam_dev/lib/python3.7/zipfile.py in _update_crc(self, newdata) 946 # Check the CRC if we're at the end of the file 947 if self._eof and self._running_crc != self._expected_crc: --> 948 raise BadZipFile("Bad CRC-32 for file %r" % self.name) 949 950 def read1(self, n): BadZipFile: Bad CRC-32 for file 'hello/0' ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844
603190621 https://github.com/pydata/xarray/issues/3815#issuecomment-603190621 https://api.github.com/repos/pydata/xarray/issues/3815 MDEyOklzc3VlQ29tbWVudDYwMzE5MDYyMQ== hmaarrfk 90008 2020-03-24T11:41:37Z 2020-03-24T11:41:37Z CONTRIBUTOR

My guess is that that xarray might be trying to write to the store character by character???

Otherwise, not too sure.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Opening from zarr.ZipStore fails to read (store???) unicode characters 573577844

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);