@jmunroe Thanks for your help here! I'm going to merge this now and take care of my remaining clean-up requests in a follow-on PR.

Just pushed a couple of commits, which should resolve the failures on Windows. It was typical int32 vs int64 NumPy on Windows nonsense.

@mrocklin are you saying that it's easier to properly rechunk data on the xarray side (as arrays) before converting to dask dataframes? That does make sense -- we have some nice structure (as multi-dimensional arrays) that is lost once the data gets put in a DataFrame.

In this case, I suppose we really should add a keyword argument like dims_order to to_dask_dataframe() that lets the user choose how they want to order dimensions on the result.

Initially, I was concerned about the resulting dask graphs when flattening out arrays in the wrong order. Although that would have bad performance implications if you need to stream the data from disk, I see now the total number of chunks no longer blows up, thanks to @pitrou's impressive rewrite of dask.array.reshape().

I think that you would want to rechunk the dask.array so that its chunks align with the outputs divisions of the dask.dataframe. For example if you have a 2d array and are partitioning along the x-axis then you will want to align the array so that there is no chunking along the y axis. In this case set_index will also be free because your data is already aligned and you already know (I think) the division values.

@jcrist @mrocklin @jhamman do any of you have opinions on my latest design question above about the order of elements in dask dataframes? Is it as important as I suspect to keep chunking/divisions consistent when converting from arrays to dataframes?

@jmunroe - can we help move this forward? I'd like to see this get into v0.10 if possible.

@jmunroe -

I added the PR checklist back to the top of this issue. The most pressing to-do item is getting some documentation written for this.

• The method will need to be added to api.rst
• We need a note briefly describing this feature in whats-new.rst
• We'll want to show an example of how this method can be used (either in the working with pandas or the dask doc sections)

@jmunroe This is great functionality -- thanks for your work on this!

One concern: if possible, I would like to avoid adding explicit dask graph building code in xarray. It looks like the canonical way to transform from a list of dask/numpy arrays to a dask dataframe is to make use of dask.dataframe.from_array along with dask.dataframe.concat: ``` In [34]: import numpy as np

In [35]: import dask.dataframe as dd

In [36]: import dask.array as da

In [37]: x = da.from_array(np.arange(5), 2)

In [38]: y = da.from_array(np.linspace(-np.pi, np.pi, 5), 2)

notice that dtype is preserved properly

In [39]: dd.concat([dd.from_array(x), dd.from_array(y)], axis=1) Out[39]: Dask DataFrame Structure: 0 1 npartitions=2 0 int64 float64 2 ... ... 4 ... ... Dask Name: concat-indexed, 26 tasks ```

Can you look into refactoring your code to make use of these?

Given that dask dataframes don't support MultiIndexes (among many other features), I have a hard time seeing them as a drop-in replacement for pandas.DataFrame. So maybe it would make sense to make this a separate method, e.g., to_dask_dataframe()?

We could also use a new method as an opportunity to slightly change the API, by not setting an index automatically. This lets us handle N-dimensional data while side-stepping the issue of MultiIndex support -- I don't think this would be very useful when limited to 1D arrays, and dask MultiIndex support seems to be a ways away (https://github.com/dask/dask/issues/1493). Also, set_index() in dask shuffles data, so it can be somewhat expensive.