home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

4 rows where author_association = "MEMBER" and issue = 569806418 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • keewis 4

issue 1

  • Dataset problem with chunk DataArray. · 4 ✖

author_association 1

  • MEMBER · 4 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
591709959 https://github.com/pydata/xarray/issues/3795#issuecomment-591709959 https://api.github.com/repos/pydata/xarray/issues/3795 MDEyOklzc3VlQ29tbWVudDU5MTcwOTk1OQ== keewis 14808389 2020-02-27T00:13:12Z 2020-02-27T00:13:12Z MEMBER

Is there a reason you need to have exactly one chunk? The chunks are used to reduce the memory needed when computing the final results and you should only need to care about them if you are trying to optimize the speed / memory footprint of your operations.

If you really need to have exactly one chunk (and you have enough memory), it might be better to use ds.load() or ds.compute() directly after the call to open_mfdataset and continue without dask.

I'm not really experienced with the backends / open_mfdataset / dask, though, so I'd appreciate input from someone with more experience.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Dataset problem with chunk DataArray. 569806418
590369650 https://github.com/pydata/xarray/issues/3795#issuecomment-590369650 https://api.github.com/repos/pydata/xarray/issues/3795 MDEyOklzc3VlQ29tbWVudDU5MDM2OTY1MA== keewis 14808389 2020-02-24T15:10:40Z 2020-02-24T15:10:40Z MEMBER

I don't think xarray.open_dataset has a parallel option so that should probably be xarray.open_mfdataset?

Sorry, I should have been clear on what I requested. You don't have to dive into the xarray code, I just wanted you to show how you use xarray so I can understand and reproduce your issue. We typically point to http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports for more explanations and the reasoning.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Dataset problem with chunk DataArray. 569806418
590316603 https://github.com/pydata/xarray/issues/3795#issuecomment-590316603 https://api.github.com/repos/pydata/xarray/issues/3795 MDEyOklzc3VlQ29tbWVudDU5MDMxNjYwMw== keewis 14808389 2020-02-24T13:21:08Z 2020-02-24T13:21:08Z MEMBER

that looks fine to me.

The strange is the DataArray chunksize changes after i modify the name 'xarray_dataarray_variable' to use another to the Dataset.

modify how? I can't find anything about xarray_dataarray_variable, so the code would be helpful.

After i modify again the chunksize to 365 to avoid the Memory Error

again, showing how you modify (in code) would help a lot

To really be able to help you and to exclude misunderstandings, I need a bit more information. To be clear, what I would like to know is: * the repr of the original dataset (is that the one from https://github.com/pydata/xarray/issues/3795#issuecomment-590294555?) * code for the operation you want to do (a description or pseudo-code is fine, too) * the repr of the data structure you get (or want to get) * and optionally, if there is something wrong with the result, a description of what you would have expected

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Dataset problem with chunk DataArray. 569806418
590290745 https://github.com/pydata/xarray/issues/3795#issuecomment-590290745 https://api.github.com/repos/pydata/xarray/issues/3795 MDEyOklzc3VlQ29tbWVudDU5MDI5MDc0NQ== keewis 14808389 2020-02-24T12:10:03Z 2020-02-24T12:10:03Z MEMBER

Could you show a code example demonstrating how you created your dataset / dataarrays and what you are trying to get? For example, something like this: ```python In [1]: import xarray as xr ...: import dask.array as da

In [2]: a = da.zeros((14610, 47, 68, 81), chunks=(365, 47, 68, 81)) ...: b = da.zeros((10950, 47, 68, 81), chunks=(365, 47, 68, 81)) ...: ds = xr.Dataset({"a": (("u", "x", "y", "z"), a), "b": (("v", "x", "y", "z"), b)}) ...: ds Out[2]: <xarray.Dataset> Dimensions: (u: 14610, v: 10950, x: 47, y: 68, z: 81) Dimensions without coordinates: u, v, x, y, z Data variables: a (u, x, y, z) float64 dask.array<chunksize=(365, 47, 68, 81), meta=np.ndarray> b (v, x, y, z) float64 dask.array<chunksize=(365, 47, 68, 81), meta=np.ndarray> ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Dataset problem with chunk DataArray. 569806418

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 15.839ms · About: xarray-datasette