issue_comments

Thank you guys for the amazing investigation into the issues here. 🤗

I was basically pinpointed at the issues from my point of view.

I have fixed the reported issues and released version 0.13.0 with the fixes included.

Third and final issue is when numpy.broadcast_to is applied to the output of zeros_like:

```

import sparse

s = sparse.COO.from_numpy([0, 0, 1, 2])

np.broadcast_to(np.zeros_like(s.todense(), shape=()), (3, ))

array([0, 0, 0])

np.broadcast_to(np.zeros_like(s, shape=()), (3, ))

ValueError: The data length does not match the coordinates given.

len(data) = 0, but 3 coords specified.

```

This seems like a genuine bug, please file it on our tracker if possible.

Third and final issue is when numpy.broadcast_to is applied to the output of zeros_like: ```

import sparse s = sparse.COO.from_numpy([0, 0, 1, 2]) np.broadcast_to(np.zeros_like(s.todense(), shape=()), (3, )) array([0, 0, 0]) np.broadcast_to(np.zeros_like(s, shape=()), (3, )) ValueError: The data length does not match the coordinates given. len(data) = 0, but 3 coords specified. ```

Ah, shape= was very recently added in 0.12.0. It wasn't there in 0.11.2.

[EDIT] It is not the only problem. Investigating further.

The second issue is that sparse.zeros_like doesn't accept the order= parameter, which is required by the same dask code linked above (it's in the kwargs in dask/wrap.py:133). This in turn triggers an unfortunate handling of TypeError on behalf of @curry, which obfuscates the exception.

da.zeros_like(a) internally invokes da.zeros(a.shape, meta=a._meta) which internally invokes np.broadcast(np.zeros_like(a._meta, shape=1), a.shape)

The problem is that sparse.zeros_like does not accept the shape= optional parameter, which is new in numpy 1.17.

This is where it gets triggered: https://github.com/dask/dask/blob/85f0b14bd36a5135ce51aeee067b6207374b00c4/dask/array/wrap.py#L128-L168

I don't think dask should write a workaround to this, and it should be just fixed upstream? CC @jrbourbeau for opinion.

Narrowed it down.

```python

import dask.array as da import sparse s = sparse.COO.from_numpy([0, 0, 1, 2]) a = da.from_array(s) z = da.zeros_like(a) z dask.array<zeros_like, shape=(4,), dtype=int64, chunksize=(4,), chunktype=sparse.COO> z.compute() <function _broadcast_trick_inner at 0x7fb8a3874e50> ``` numpy-1.20.1 dask-2021.3.0 sparse-0.11.2

Would it be possible to have the two issues tracked by separate tickets?

definitely, let's keep this one focused on sparse. I think removing the label should be enough to have CI open a new issue (not sure, though).

I see now. I got really confused by the matplotlib issue which is what the opening post of this ticket is about. Would it be possible to have the two issues tracked by separate tickets?

The failure in xarray/tests/test_sparse.py::test_chunk doesn't appear anywhere in recent CI runs

It was xfailed in #5729 , maybe that explains it

I'm a bit lost. The failure in xarray/tests/test_sparse.py::test_chunk doesn't appear anywhere in recent CI runs and I can't reproduce it locally.

The ongoing failures in upstream-dev: FAILED xarray/tests/test_plot.py::TestFacetGrid::test_can_set_norm FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_line_plot FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_pcolormesh_plot FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_contour_plot Aren't in any way related to either sparse or dask; they appear when I upgrade matplotlib from 3.4.3 to 3.5.0b1.

FWIW, the upstream-dev CI installs sparse=0.12.0+55.ge706c2e (i.e. the most recent state of the repository). However, since our other CI is also failing I guess this does not change much.

We already require pandas>=1.0 and I guess we can bump numpy (NEP29 allows it for releases after Jul 26), but I'm not sure why that would be an issue for the upstream-dev CI. For reference, the installed versions arenumpy=1.22.0.dev0+716.g20807c743 and pandas=1.4.0.dev0+436.gfcf3f24e02.

~Interestingly, the installation of dask from github started failing two days ago (see the last succeeding and the first failing install – unless I'm misinterpreting the logs?). Any idea how to fix that?~ The most recent run correctly installed dask so I guess this was a temporary issue (or dask resolved this really quickly)

This looks like dask.

```python ______ test_chunk ________

@requires_dask
def test_chunk():
    s = sparse.COO.from_numpy(np.array([0, 0, 1, 2]))
    a = DataArray(s)
    ac = a.chunk(2)
    assert ac.chunks == ((2, 2),)
    assert isinstance(ac.data._meta, sparse.COO)

  assert_identical(ac, a)

self = <COO: shape=(2,), dtype=bool, nnz=0, fill_value=False> other = <function _broadcast_trick_inner at 0x7f325aef91f0>

def func(self, other):
    if _disables_array_ufunc(other):
        return NotImplemented

  return ufunc(other, self)

E TypeError: operand type(s) all returned NotImplemented from array_ufunc(<ufunc 'bitwise_and'>, 'call', <function _broadcast_trick_inner at 0x7f325aef91f0>, <COO: shape=(2,), dtype=bool, nnz=0, fill_value=False>): 'curry', 'COO' ```

cc @jrbourbeau

This is showing up in the normal CI now, see https://github.com/pydata/xarray/pull/5667/checks?check_run_id=3329247566

this seems to be caused by changes to either dask or sparse, not sure which.