issue_comments

Table actions

7 rows where user = 34686298 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, reactions, created_at (date), updated_at (date)

user 1

  • peterdudfield · 7 ✖

author_association 1

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1453901550 https://github.com/pydata/xarray/issues/4122#issuecomment-1453901550 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85WqMbu peterdudfield 34686298 2023-03-03T18:03:49Z 2023-03-03T18:03:49Z NONE

I never needed to specify an engine when writing, you only need it when reading the file. I use the engine="scipy" one for reading.

using engine="scipy" worked - thank you

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453899322 https://github.com/pydata/xarray/issues/4122#issuecomment-1453899322 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85WqL46 peterdudfield 34686298 2023-03-03T18:02:04Z 2023-03-03T18:02:04Z NONE

I use the engine="scipy" one for reading.

This is netCDF3, in that case. If that's fine for you, no problem.

What do you mean this is netcdf3?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453562756 https://github.com/pydata/xarray/issues/4122#issuecomment-1453562756 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85Wo5uE peterdudfield 34686298 2023-03-03T13:52:08Z 2023-03-03T17:48:15Z NONE

Maybe it is netCDF3? xarray is supposed to be able to determine the file type

with fsspec.open("s3://some_bucket/some_remote_destination.nc", mode="rb") as ff: ds = xr.open_dataset(ff)

but maybe play with the engine= argument.

Thanks, i tried to make sure it was engine=h5netcdf when saving, but not sure this worked

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453883439 https://github.com/pydata/xarray/issues/4122#issuecomment-1453883439 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85WqIAv peterdudfield 34686298 2023-03-03T17:47:43Z 2023-03-03T17:47:43Z NONE

It could be here's some running notes of mine - https://github.com/openclimatefix/MetOfficeDataHub/issues/65

The same method is python with fsspec.open("simplecache::s3://nowcasting-nwp-development/data/test.netcdf", mode="wb") as f: dataset.to_netcdf(f,engine='h5netcdf')

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1453553088 https://github.com/pydata/xarray/issues/4122#issuecomment-1453553088 https://api.github.com/repos/pydata/xarray/issues/4122 IC_kwDOAMm_X85Wo3XA peterdudfield 34686298 2023-03-03T13:43:58Z 2023-03-03T13:45:59Z NONE

What didn't work:

python f = fsspec.filesystem("s3", anon=False) with f.open("some_bucket/some_remote_destination.nc", mode="wb") as ff: xr.open_dataset("some_local_file.nc").to_netcdf(ff)

this results in a OSError: [Errno 29] Seek only available in read mode exception

Changing the above to

python with fsspec.open("simplecache::s3://some_bucket/some_remote_destination.nc", mode="wb") as ff: xr.open_dataset("some_local_file.nc").to_netcdf(ff)

fixed it.

How would you go about reading this file? Once it is saved in s3

Im currently getting an error ValueError: b'CDF\x02\x00\x00\x00\x00' is not the signature of a valid netCDF4 file

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Document writing netcdf from xarray directly to S3 631085856
1352938631 https://github.com/pydata/xarray/issues/7354#issuecomment-1352938631 https://api.github.com/repos/pydata/xarray/issues/7354 IC_kwDOAMm_X85QpDSH peterdudfield 34686298 2022-12-15T11:41:21Z 2022-12-15T11:41:21Z NONE

Thanks @jhamman for looking into this. I'll try your suggestions

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'open_mfdataset' zarr zip timestamp issue 1474785646
1346141641 https://github.com/pydata/xarray/issues/7354#issuecomment-1346141641 https://api.github.com/repos/pydata/xarray/issues/7354 IC_kwDOAMm_X85QPH3J peterdudfield 34686298 2022-12-12T09:16:44Z 2022-12-12T09:16:44Z NONE

@peterdudfield - have you tried this workflow with the latest version of xarray (2022.12.0)?

Yea the same bug appeared. So this appears at 2022.6.0 and onwards. 

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   'open_mfdataset' zarr zip timestamp issue 1474785646

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);