issue_comments

Table actions

2 rows where author_association = "CONTRIBUTOR", issue = 449706080 and user = 868027 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • DocOtak · 2 ✖

issue 1

  • Remote writing NETCDF4 files to Amazon S3 · 2 ✖

author_association 1

  • CONTRIBUTOR · 2 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
497066189 https://github.com/pydata/xarray/issues/2995#issuecomment-497066189 https://api.github.com/repos/pydata/xarray/issues/2995 MDEyOklzc3VlQ29tbWVudDQ5NzA2NjE4OQ== DocOtak 868027 2019-05-29T18:56:17Z 2019-05-29T18:56:17Z CONTRIBUTOR

Thanks @rabernat I had forgotten about the other netcdf storage engines... do you know if h5netcdf stable enough that I should use in "production" outside of xarray for my netcdf4 reading/writing needs?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Remote writing NETCDF4 files to Amazon S3 449706080
497026828 https://github.com/pydata/xarray/issues/2995#issuecomment-497026828 https://api.github.com/repos/pydata/xarray/issues/2995 MDEyOklzc3VlQ29tbWVudDQ5NzAyNjgyOA== DocOtak 868027 2019-05-29T17:11:10Z 2019-05-29T17:12:51Z CONTRIBUTOR

Hi @Non-Descript-Individual

I've found that the netcdf4-python library really wants to have direct access to a disk/filesystem to work, it also really wants to do its own file access management. I've always attributed this to the python library being a wrapper for the netcdf C library.

My guess would be that the easiest way to do what you want is to separate the writing of the netcdf file step in xarray from the putting the file into S3. Something like this:

python x.to_netcdf('temp_file.nc') s3.upload_file('temp_file.nc', 'bucketname', 'real_name_for_temp_file.nc')

The netcdf4-python library does seem to provide an interface for the "diskless" flags. In this case, from the examples it looks to give you a bunch of bytes in a memoryview object on calling close(). I'm not sure this is accessible from xarray though.

Alternatively, @rabernat is an advocate of using zarr when putting netcdf compatible data into cloud storage, the zarr docs provide an example using s3fs

Quick edit: Here is the to_zarr docs in xarray

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Remote writing NETCDF4 files to Amazon S3 449706080

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);