home / github

Menu
  • GraphQL API
  • Search all tables

issue_comments

Table actions
  • GraphQL API for issue_comments

8 rows where author_association = "MEMBER" and issue = 363299007 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

user 5

  • TomNicholas 3
  • keewis 2
  • rabernat 1
  • dcherian 1
  • spencerkclark 1

issue 1

  • save "encoding" when using open_mfdataset · 8 ✖

author_association 1

  • MEMBER · 8 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
610073334 https://github.com/pydata/xarray/issues/2436#issuecomment-610073334 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDYxMDA3MzMzNA== keewis 14808389 2020-04-06T22:38:30Z 2020-04-06T22:38:30Z MEMBER

I removed it since it doesn't change anything.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
610053703 https://github.com/pydata/xarray/issues/2436#issuecomment-610053703 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDYxMDA1MzcwMw== keewis 14808389 2020-04-06T21:45:47Z 2020-04-06T22:03:08Z MEMBER

unfortunately, numpy does not allow us to put cftime object into dtypes (yet!), so ds.time.values is a numpy.ndarray with dtype object, containing cftime objects. To make the code work, use it with ds.time.values[0]. Of course, that won't help if the array contains objects of more than one type.

```python

import cftime isinstance(ds.time.values[0], cftime.DatetimeNoLeap) True type(ds.time.values[0]) <class 'cftime._cftime.DatetimeNoLeap'> ```

In #3498, the original proposal was to name the new kwarg master_file, but later it was renamed to attrs_file. If l_f is a list of file paths, you used it correctly.

Before trying to help with debugging your issue: could you post the output of xr.show_versions()? That would help narrowing down on whether it's a dependency issue or a bug in xarray.

Also, could you try to demonstrate your issue using a synthetic example? I've been trying to reproduce it with: python In [14]: units = 'days since 2000-02-25' ...: times = cftime.num2date(np.arange(7), units=units, calendar='365_day') ...: for x in range(5): ...: ds = xr.DataArray( ...: np.arange(x, 7 + x).reshape(7, 1), ...: coords={"time": times, "x": [x]}, ...: dims=['time', "x"], ...: name='a', ...: ).to_dataset() ...: ds.to_netcdf(f'data-noleap{x}.nc') ...: paths = sorted(glob.glob("data-noleap*.nc")) ...: with xr.open_mfdataset(paths, combine="by_coords") as ds: ...: print(ds.time.encoding) ...: {'zlib': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': '.../data-noleap0.nc', 'original_shape': (7,), 'dtype': dtype('int64'), 'units': 'days since 2000-02-25 00:00:00.000000', 'calendar': 'noleap'}

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
610060062 https://github.com/pydata/xarray/issues/2436#issuecomment-610060062 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDYxMDA2MDA2Mg== dcherian 2448579 2020-04-06T22:01:13Z 2020-04-06T22:01:13Z MEMBER

This example works without attrs_file specified.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
610008589 https://github.com/pydata/xarray/issues/2436#issuecomment-610008589 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDYxMDAwODU4OQ== TomNicholas 35968931 2020-04-06T20:05:10Z 2020-04-06T20:05:10Z MEMBER

@TomNicholas I forgot about this sorry.

No worries!

I just made a quick check with the latest xarray master and I still have the problem ... see code.

3498 added a new keyword argument to open_mfdataset, to choose which file to load to attributes from, can you try using that?

time.encoding is empty while it is as expected when opening any of the files with open_dataset instead

If this is the case, then to solve your original problem, you could also try using the preprocess argument to open_mfdataset to store the encoding somewhere where it won't be lost? i.e.

```python def store_encoding(ds): encoding = ds['time'].encoding ds.time.attrs['calendar_encoding'] = encoding return ds

snw = xr.open_mfdataset(l_f, combine='nested', concat_dim='time', master_file=lf[0], preprocess=store_encoding)['snw'] ```

Related question but maybe out of line, is there any way to know that the snw.time type is cftime.DatetimeNoLeap (as it is visible in the overview of snw.time)?

I'm not familiar with these classes, but presumably you mean more than just checking with isinstance()? e.g. python from cftime import DatetimeNoLeap print(isinstance(snw.time.values, cftime.DatetimeNoLeap))

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
609479629 https://github.com/pydata/xarray/issues/2436#issuecomment-609479629 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDYwOTQ3OTYyOQ== TomNicholas 35968931 2020-04-05T20:44:00Z 2020-04-05T20:44:00Z MEMBER

@sbiner I know it's been a while, but I expect that #3498 and #3877 probably resolve your issue?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
449737841 https://github.com/pydata/xarray/issues/2436#issuecomment-449737841 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDQ0OTczNzg0MQ== TomNicholas 35968931 2018-12-24T14:02:57Z 2018-12-24T14:02:57Z MEMBER

If open_mfdataset() is actually dropping the encoding, then this is an issue related to #1614. That's because in open_mfdataset() while the attrs are explicitly set to those of the first supplied dataset, I don't see any similar explicit treatment of the encoding. I think that means the encoding is being set by what happens inside the core of auto_combine(), and is presumably being lost upon some of the concat or merge operations which happen inside auto_combine().

So I think to fix this then either open_mfdataset() should contain explicit treatment of the encoding, or the rules for propagating the encoding through the auto_combine() should be solidified.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
424399958 https://github.com/pydata/xarray/issues/2436#issuecomment-424399958 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDQyNDM5OTk1OA== spencerkclark 6628425 2018-09-25T15:54:28Z 2018-09-25T15:54:28Z MEMBER

@sbiner are you looking at the encoding attribute of the full Dataset or the time variable? The time variable should retain the calendar encoding (the Dataset will not). E.g.: ``` In [1]: import cftime

In [2]: import numpy as np

In [3]: import xarray as xr

In [4]: units = 'days since 2000-02-25'

In [5]: times = cftime.num2date(np.arange(7), units=units, calendar='365_day')

In [6]: da = xr.DataArray(np.arange(7), coords=[times], dims=['time'], name='a')

In [7]: da.to_netcdf('data-noleap.nc')

In [8]: ds = xr.open_dataset('data-noleap.nc')

In [9]: ds.encoding['calendar']

KeyError Traceback (most recent call last) <ipython-input-38-677c245c7bb8> in <module>() ----> 1 default.encoding['calendar']

KeyError: 'calendar'

In [10]: ds.time.encoding['calendar'] Out[10]: u'noleap' ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007
424111082 https://github.com/pydata/xarray/issues/2436#issuecomment-424111082 https://api.github.com/repos/pydata/xarray/issues/2436 MDEyOklzc3VlQ29tbWVudDQyNDExMTA4Mg== rabernat 1197350 2018-09-24T20:20:04Z 2018-09-24T20:20:04Z MEMBER

Do you know you can access them in the .encoding namespace? Is that not sufficient for your needs?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save "encoding" when using open_mfdataset 363299007

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 15.391ms · About: xarray-datasette
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows