issue_comments

OK, thanks for clarify with that example.

I think we can track down the issue to the result of b.sum(dim='foo'): ```

b.data <COO: shape=(3, 6), dtype=float64, nnz=18, fill_value=nan> b.sum(dim='foo').data <COO: shape=(6,), dtype=float64, nnz=6, fill_value=0.0> ```

The fill value here is actually arbitrary, since the array is entirely dense. If this fill value were still nan, the later operation combining these arrays would work.

That said, sparse is making a reasonable choice here: nansum() applied to an array with all values given by nan is 0. Unless sparse wants to add special logic for handling arrays with different sparsities, I don't know how they could change this.

Options for dealing with this: - Use .mean() instead of .sum(). - Explicitly convert c into a dense array before combining it, e.g., d = xr.concat([b, c.copy(data=c.data.todense())], dim='foo') (syntax could be better). But this currently errors with: ValueError: All arrays must be instances of SparseArray. from sparse. Maybe sparse's concatenate could be updated to handle the mixed ndarray/sparse case? - Add some ergonomic way to explicitly override fill_value on sparse data in xarray, e.g., b.sum(dim='foo').with_fill_value(np.nan). - In principle, b.sum(dim='foo', min_count=1) could return a sparse array with fill_value=nan, but currently it doesn't.

Sparse only lets you combine arrays with different fill values if the result would also have a fixed value. That's why you can multiply or add but not concatenate, e.g., assert x.fill_value == 1 and y.fill_value == 2 assert (x + y).fill_value == 3 assert (x * y).fill_value == 2 np.stack([x, y]) # error, would need a mixture of different fill values to represent

Multiple fill values simply aren't representable by sparse's data model.

I think you could work this by wrapping the sparse arrays in dask first.