home / github

Menu
  • GraphQL API
  • Search all tables

issue_comments

Table actions
  • GraphQL API for issue_comments

7 rows where issue = 253136694 and user = 2443309 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • jhamman · 7 ✖

issue 1

  • WIP: Zarr backend · 7 ✖

author_association 1

  • MEMBER 7
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
364802374 https://github.com/pydata/xarray/pull/1528#issuecomment-364802374 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDM2NDgwMjM3NA== jhamman 2443309 2018-02-11T23:54:01Z 2018-02-11T23:54:01Z MEMBER

@martindurant - If I understand your question correctly, I think you should be able to follow a pretty standard xarray workflow:

```Python ds = xr.Dataset() ds['your_varname'] = xr.DataArray(some_dask_array, dims=['dimname0', 'dimname1', ...], coords=dict_of_preknown_coords)

repeat for each variable you want in your dataset

ds.to_zarr(some_zarr_store)

then to open

ds2 = xr.open_zarr(some_zarr_store) ```

Two things to note:

1) if you are looking for decent performance when writing to a remote store, make sure you're working off xarray@master as #1800 fixed a number of choke points in the to_zarr implementation 2) if you are pushing to GCS, some_zarr_store can be a GCSMap.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694
349738624 https://github.com/pydata/xarray/pull/1528#issuecomment-349738624 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDM0OTczODYyNA== jhamman 2443309 2017-12-06T18:54:41Z 2017-12-06T18:54:56Z MEMBER

@rabernat - in trying out your branch, I've run into this error (mentioned by @mrocklin in pangeo-data/pangeo#19):

```Python-traceback ... ~/anaconda/envs/pangeo-dev/lib/python3.6/site-packages/xarray-0.10.0_79_g7b50320-py3.6.egg/xarray/backends/zarr.py in _extract_zarr_variable_encoding(variable, raise_on_invalid) 228 229 chunks = _determine_zarr_chunks(encoding.get('chunks'), variable.chunks, --> 230 variable.ndim) 231 encoding['chunks'] = chunks 232 return encoding

~/anaconda/envs/pangeo-dev/lib/python3.6/site-packages/xarray-0.10.0_79_g7b50320-py3.6.egg/xarray/backends/zarr.py in _determine_zarr_chunks(enc_chunks, var_chunks, ndim) 134 "Zarr requires uniform chunk sizes excpet for final chunk." 135 " Variable %r has incompatible chunks. Consider " --> 136 "rechunking using chunk()." % var_chunks) 137 # last chunk is allowed to be smaller 138 last_var_chunk = all_var_chunks[-1]

TypeError: not all arguments converted during string formatting ```

As far as I can tell, reworking my chunk sizes to divide evenly into the dataset dimensions has corrected the problem.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694
348414545 https://github.com/pydata/xarray/pull/1528#issuecomment-348414545 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDM0ODQxNDU0NQ== jhamman 2443309 2017-12-01T06:40:47Z 2017-12-01T06:40:47Z MEMBER

@rabernat - following @shoyer's thoughts here and in #1753, I'm not apposed to skipping the last few failing tests and live to fight strings another day.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694
345128506 https://github.com/pydata/xarray/pull/1528#issuecomment-345128506 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDM0NTEyODUwNg== jhamman 2443309 2017-11-17T02:38:41Z 2017-11-17T02:38:41Z MEMBER

@rabernat - It might a little but we'll sort it out. See https://github.com/rabernat/xarray/pull/3.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694
345026224 https://github.com/pydata/xarray/pull/1528#issuecomment-345026224 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDM0NTAyNjIyNA== jhamman 2443309 2017-11-16T18:53:42Z 2017-11-16T18:53:42Z MEMBER

@rabernat - FYI: I'm playing with your branch a bit today.

@shoyer and @rabernat, can we brainstorm what a ZarrArrayWraper would need to be compatible with the new indexing API? I'm happy to implement it but could use a few pointers to get started.

```Python class ZarrArrayWraper(BackendArray): def init(self, variable_name, datastore): self.datastore = datastore self.variable_name = variable_name array = self.get_array() self.shape = array.shape self.dtype = np.dtype(array.dtype.kind + str(array.dtype.itemsize))

def get_array(self):
    self.datastore.assert_open()
    return self.datastore.ds[self.variable_name]  # returns a zarr-array

def __getitem__(self, key):
    with self.datastore.ensure_open(autoclose=True):
        data = IndexingAdapter(self.get_array())[key]  # which indexing adapter? 
        return np.array(data, dtype=self.dtype, copy=copy)

```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694
334316122 https://github.com/pydata/xarray/pull/1528#issuecomment-334316122 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDMzNDMxNjEyMg== jhamman 2443309 2017-10-04T23:14:58Z 2017-10-04T23:14:58Z MEMBER

@rabernat - testing should be fully functional now.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694
333579128 https://github.com/pydata/xarray/pull/1528#issuecomment-333579128 https://api.github.com/repos/pydata/xarray/issues/1528 MDEyOklzc3VlQ29tbWVudDMzMzU3OTEyOA== jhamman 2443309 2017-10-02T15:58:05Z 2017-10-02T15:58:05Z MEMBER

@rabernat - re backends testing, #1557 is pretty close. I can wrap it up this week.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  WIP: Zarr backend 253136694

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 53.232ms · About: xarray-datasette