issue_comments

Table actions

5 rows where author_association = "NONE", issue = 631085856 and user = 34686298 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • peterdudfield · 5 ✖

issue 1

  • Document writing netcdf from xarray directly to S3 · 5 ✖

author_association 1

  • NONE · 5 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1453901550 https://github.com/pydata/xarray/issues/4122#issuecomment-1453901550 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85WqMbu peterdudfield 34686298 2023-03-03T18:03:49Z 2023-03-03T18:03:49Z NONE

I never needed to specify an engine when writing, you only need it when reading the file. I use the engine="scipy" one for reading.

using engine="scipy" worked - thank you

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453899322 https://github.com/pydata/xarray/issues/4122#issuecomment-1453899322 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85WqL46 peterdudfield 34686298 2023-03-03T18:02:04Z 2023-03-03T18:02:04Z NONE

I use the engine="scipy" one for reading.

This is netCDF3, in that case. If that's fine for you, no problem.

What do you mean this is netcdf3?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453562756 https://github.com/pydata/xarray/issues/4122#issuecomment-1453562756 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85Wo5uE peterdudfield 34686298 2023-03-03T13:52:08Z 2023-03-03T17:48:15Z NONE

Maybe it is netCDF3? xarray is supposed to be able to determine the file type

with fsspec.open("s3://some_bucket/some_remote_destination.nc", mode="rb") as ff: ds = xr.open_dataset(ff)

but maybe play with the engine= argument.

Thanks, i tried to make sure it was engine=h5netcdf when saving, but not sure this worked

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453883439 https://github.com/pydata/xarray/issues/4122#issuecomment-1453883439 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85WqIAv peterdudfield 34686298 2023-03-03T17:47:43Z 2023-03-03T17:47:43Z NONE

It could be here's some running notes of mine - https://github.com/openclimatefix/MetOfficeDataHub/issues/65

The same method is python with fsspec.open("simplecache::s3://nowcasting-nwp-development/data/test.netcdf", mode="wb") as f: dataset.to_netcdf(f,engine='h5netcdf')

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453553088 https://github.com/pydata/xarray/issues/4122#issuecomment-1453553088 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85Wo3XA peterdudfield 34686298 2023-03-03T13:43:58Z 2023-03-03T13:45:59Z NONE

What didn't work:

python f = fsspec.filesystem("s3", anon=False) with f.open("some_bucket/some_remote_destination.nc", mode="wb") as ff: xr.open_dataset("some_local_file.nc").to_netcdf(ff)

this results in a OSError: [Errno 29] Seek only available in read mode exception

Changing the above to

python with fsspec.open("simplecache::s3://some_bucket/some_remote_destination.nc", mode="wb") as ff: xr.open_dataset("some_local_file.nc").to_netcdf(ff)

fixed it.

How would you go about reading this file? Once it is saved in s3

Im currently getting an error ValueError: b'CDF\x02\x00\x00\x00\x00' is not the signature of a valid netCDF4 file

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);