issue_comments

Table actions

9 rows where issue = 28375178 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

author_association 3

issue 1

  • Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) · 9 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
776721835 https://github.com/pydata/xarray/issues/23#issuecomment-776721835 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDc3NjcyMTgzNQ== pikulmar 64951363 2021-02-10T13:56:24Z 2021-02-10T13:56:24Z NONE

Still an issue, as far as I can tell. Possibly duplicated in https://github.com/pydata/xarray/issues/3372.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
748611575 https://github.com/pydata/xarray/issues/23#issuecomment-748611575 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDc0ODYxMTU3NQ== stale[bot] 26384082 2020-12-20T14:00:16Z 2020-12-20T14:00:16Z NONE

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
454281015 https://github.com/pydata/xarray/issues/23#issuecomment-454281015 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDQ1NDI4MTAxNQ== shoyer 1217238 2019-01-15T06:27:00Z 2019-01-15T06:27:00Z MEMBER

This is actually finally possible to support now with h5py, which as of the latest release supports reading/writing to file-like objects in Python.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
454162987 https://github.com/pydata/xarray/issues/23#issuecomment-454162987 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDQ1NDE2Mjk4Nw== max-sixty 5635139 2019-01-14T21:12:07Z 2019-01-14T21:12:07Z MEMBER

In an effort to reduce the issue backlog, I'll close this, but please reopen if you disagree

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
90780331 https://github.com/pydata/xarray/issues/23#issuecomment-90780331 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDkwNzgwMzMx shoyer 1217238 2015-04-08T02:07:31Z 2015-04-08T02:07:31Z MEMBER

Just wrote a little library to do netCDF4 via h5py: https://github.com/shoyer/h5netcdf

Unfortunately h5py still can't do in-memory file images (https://github.com/h5py/h5py/issues/552). But it does give an alternative way to read/write netCDF4 without going via the Unidata libraries. There is experimental support for engine='h5netcdf' in my dask PR: https://github.com/xray/xray/pull/381

pytables was not a viable option because it can't read or write HDF5 dimension scales, which are necessary for dimensions in netCDF4 files.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
36186835 https://github.com/pydata/xarray/issues/23#issuecomment-36186835 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDM2MTg2ODM1 shoyer 1217238 2014-02-26T22:38:42Z 2014-02-26T22:38:42Z MEMBER

HDF5 supports homogeneous n-dimensional arrays and metadata, which in principle should be all we need. Actually, under the covers netCDF4 is HDF5. But yes, we would have to do some work to reinvent this.

On Wed, Feb 26, 2014 at 2:32 PM, ebrevdo notifications@github.com wrote:

Looks like this may be the only option. Based on my tests, netCDF4 is strongly antithetical to any kind of streams/piped buffers. If we go the hdf5 route, we'd have to reimplement the CDM/netcdf4 on top of hdf5, no?

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/23#issuecomment-36186205 .

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
36186205 https://github.com/pydata/xarray/issues/23#issuecomment-36186205 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDM2MTg2MjA1 ebrevdo 1794715 2014-02-26T22:32:06Z 2014-02-26T22:32:06Z CONTRIBUTOR

Looks like this may be the only option. Based on my tests, netCDF4 is strongly antithetical to any kind of streams/piped buffers. If we go the hdf5 route, we'd have to reimplement the CDM/netcdf4 on top of hdf5, no?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
36185024 https://github.com/pydata/xarray/issues/23#issuecomment-36185024 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDM2MTg1MDI0 akleeman 514053 2014-02-26T22:21:01Z 2014-02-26T22:21:01Z CONTRIBUTOR

Another similar option would be to use in-memory HDF5 objects for which Todd Small found an option:

Writing to a string:

h5_file = tables.open_file("in-memory", title=my_title, mode="w", 12 driver="H5FD_CORE", driver_core_backing_store=0) ... [add variables] ... image = h5_file.get_file_image()

Reading from a string

h5_file = tables.open_file("in-memory", mode="r", driver="H5FD_CORE", driver_core_image=image, driver_core_backing_store=0)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178
36184762 https://github.com/pydata/xarray/issues/23#issuecomment-36184762 https://api.github.com/repos/pydata/xarray/issues/23 MDEyOklzc3VlQ29tbWVudDM2MTg0NzYy shoyer 1217238 2014-02-26T22:18:28Z 2014-02-26T22:18:28Z MEMBER

Another option is to add an HDF5 backend with pytables. @ToddSmall has a demo script somewhere that shows how you can pass around in-memory HDF5 objects between processes.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Cross-platform in-memory serialization of netcdf4 (like the current scipy-based dumps) 28375178

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);