issues

Table actions

3 rows where user = 29051639 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: comments, created_at (date), updated_at (date), closed_at (date)

type 2

state 2

repo 1

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
758606082 MDExOlB1bGxSZXF1ZXN0NTMzNzUyNDEy 4659 xr.DataArray.from_dask_dataframe feature AyrtonB 29051639 open 0     18 2020-12-07T15:22:52Z 2023-06-08T20:25:58Z   CONTRIBUTOR   0 pydata/xarray/pulls/4659

This feature allows users to convert Dask DataFrames (of a single type) into a DataArray that uses a Dask array. This solves a gap in the Python ecosystem around saving Dask DataFrames to Zarr which is currently not possible without loading the full dataset into memory.

This feature specifically handles the case where a DataFrame is of a single type, a xr.Dataset.from_dask_dataframe could be developed in future to handle the multi-type case.

  • [x] Closes #4650, closes #3929
  • [x] Tests added
  • [x] Passes isort . && black . && mypy . && flake8
  • [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
  • [x] New functions/methods are listed in api.rst
 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4659/reactions",
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     xarray 13221727 pull
757751542 MDExOlB1bGxSZXF1ZXN0NTMzMDcxODYy 4653 corrected a minor spelling mistake AyrtonB 29051639 closed 0     4 2020-12-05T18:16:41Z 2020-12-05T19:18:10Z 2020-12-05T19:03:27Z CONTRIBUTOR   0 pydata/xarray/pulls/4653

What it says on the tin, changed xarrray.open_dataset to xarray.open_dataset.

The mistake was in documentation so no changes to any code have been made

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4653/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     xarray 13221727 pull
757660307 MDU6SXNzdWU3NTc2NjAzMDc= 4650 Ability to Pass Dask Arrays as `data` in DataArray Creation AyrtonB 29051639 closed 0     4 2020-12-05T11:33:03Z 2020-12-05T18:23:11Z 2020-12-05T13:13:10Z CONTRIBUTOR      

Is your feature request related to a problem? Please describe.

I'm trying to convert a dask dataframe into a dask xarray without having to load the data fully into memory.

I was hoping I'd be able to pass df.values which is a Dask array to the data parameter in xr.DataArray

```python idx_dim = 'datetime' col_dim = 'fueltypes'

xr.DataArray(df.values, [df.index, df.columns], [idx_dim, col_dim]) ```

However this raises the error: ValueError: conflicting sizes for dimension 'datetime': length nan on the data but length 90386 on coordinate 'datetime'


Describe the solution you'd like

An ability to create DataArrays from dask dataframes, similar to the existing reverse method for converting Datasets to dask dataframes: Dataset.to_dask_dataframe


Describe alternatives you've considered

I tried using xr.Dataset.from_dataframe(df) but it required the dataframe to be fully loaded into memory

Additionally, unlike the standard Pandas dataframe the Dask dataframe does not have a .to_xarray method.


Additional context

This is in part made necessary by the decision of the Zarr developers to not support saving of dask dataframes to zarr, instead suggesting that you convert to an xarray and then save that to zarr.

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4650/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);