issue_comments

If we're doing to do this, I would suggest that the right signature is xarray.map(func, *datasets, **optional_kwargs), matching Python's builtin map.

What I'd like to ensure is a clean separation between the arguments of xarray.map and func. Map has three "own" parameters, like func, datasets and keep_attrs. By using the **kwargs approach we are excluding these parameter names from func. Not saying that is likely that anyone would apply a function with such parameter names. But not impossible either. Also having a real dict for keyword args (and maybe a list for positional arguments of func) is more explicit.

In my implementation the order of parameters is datasets and then func to match that of Dataset.map with the implicit self. But probably func and then the datasets is more intuitive.

* Are there many other cases outside of `xr.dot` which only operate on `DataArray`s? If not, we could update that function to take a `Dataset`

I think it would be a good idea to extend dot to Datasets. However a user may wish to map a custom DataArray function to Dataset.

* Maybe jumping ahead — are there functions where the result of `func(ds1, ds2)` shouldn't be that function mapped over the matching variables?

Not sure of the context of this. In the most general case one can certainly implement any function on ds1 and ds2. Or are you referring to the built-ins such as .dot?

Could I ask what the common use cases for this would be? If I understand correctly, running map(x, y, lambda x: x + y) is equivalent to x + y.

The motivating use case was that I wanted to compute the dot-product of two DataSets (=all of their matching variables). But in general any other function which is not as simple as x + y could be used here.

I've reimplemented from_dataframe to make use of in #4184, and it indeed makes things much, much faster! The original example in this thread is now 40x faster.

Very good news! Thanks for implementing it!

@AndrewWilliams3142 I see. Thanks.

Well, actually I was thinking, that correcting it for someone who is working on the code on a daily basis is ~30 seconds. For me, I think, it would be quite a bit of overhead for a single character...

Just a small comment: in the docs (http://xarray.pydata.org/en/latest/generated/xarray.cov.html#xarray.cov) there is a typo: da_a is declared twice, the second should really be da_b.

I encountered this bug a few days ago. I understand it isn't trivial to fix, but would it be possible to check and throw an exception? Still better than having it go unnoticed. Thanks

I know this is not a recent thread but I found no resolution, and we just ran in the same issue recently. In our case we had a pandas series of roughly 15 milliion entries, with a 3-level multi-index which had to be converted to an xarray.DataArray. The .to_xarray took almost 2 minutes. Unstack + to_array took it down to roughly 3 seconds, provided the last level of the multi index was unstacked.

However a much faster solution was through numpy array. The below code is based on the idea of Igor Raush

(In this case df is a dataframe with a single column, or a series) arr = np.full(df.index.levshape, np.nan) arr[tuple(df.index.codes)] = df.values.flat da = xr.DataArray(arr,dims=df.index.names,coords=dict(zip(df.index.names, df.index.levels)))

Is it already decided what the resolution should be? * Giving a warning, as the title of this thread suggests? * Disable setting .values directly for dimensions? * Or making sure that .indexes are updated when .values are set directly

I now came across this issue, which still seems to be open. Are the statements made earlier still valid? Are there any concrete plans maybe to fix this in the near future?

Actually I just saw that the requirement for xarray 0.8.2 is: pandas >= 0.15.0, I don't know whether it is possible to specify: 0.19.1>=pandas>=0.15.0.

Just ran into this issue when wanted to install packages for some newcomers in our company.

But actually we solved the issue by adding strict version requirement for both xarray and pandas. (We need these versions because we have some pickled files using these formats which we need to read. Until I find the time to get rid of them.)