issue_comments

Thanks @dcherian. Feel free to let me know if you or others run into this issue elsewhere. I also think it's an edge case the vast majority of users won't encounter (this is the first occurrence I've seen outside of the dask test suite), but if that turns out to not be the case then we can make adjustments upstream in dask 👍

Just checking in, @dcherian does this look okay to you? Happy to continue iterating if you see a problem with just removing this config option

I've not looked super deeply, so please let me know if I'm missing something, but I think that these lines in get_scheduler are the relevant ones here

https://github.com/dask/dask/blob/db5b2178a79cacc1c882d60a82bf86e2e188eccb/dask/base.py#L1405-L1406

Previously, distributed would set the scheduler config option to point to the default distributed.Client (if one existed). We've since changed that logic and distributed no no longer uses the config option for saying "you've got a distributed.Client, you should use it".

I think the problem here is, with the previous config-based behavior, the scheduler option, which is currently being set to the single-threaded scheduler in the test suite, would be overwritten by distributed to point to the Client. However, now that we're no longer using the scheduler config option in distributed, the test suite is actually using the single-threaded scheduler, which is why get_scheduler() is returning get_sync when using the latest dask / distributed release.

I'd argue the new behavior is actually what we want, but I see what you're saying about it being a change in behavior. I think in this case though it's just a tests-related issue. Does that sounds right? Or was setting the scheduler config option important for real-life user code too?

Hmm I'm not able to reproduce with this example using the latest 2023.1.1 release:

```python In [1]: import dask

In [2]: dask.utils.show_versions() { "Python": "3.10.4", "Platform": "Darwin", "dask": "2023.1.1+3.gdb5b2178a", "distributed": "2023.1.1", "numpy": "1.24.1", "pandas": "2.0.0.dev0+1309.g7f2aa8f46a", "cloudpickle": "2.0.0", "fsspec": "2023.1.0+5.g012816b", "bokeh": "2.4.3", "fastparquet": "2022.12.1.dev6", "pyarrow": "11.0.0.dev316", "zarr": "2.4.1.dev528" } In [3]: from distributed import Client

In [4]: type(dask.base.get_scheduler()) Out[4]: NoneType

In [5]: c = Client()

In [6]: dask.base.get_scheduler() Out[6]: <bound method Client.get of \<Client: 'tcp://127.0.0.1:61659' processes=4 threads=8, memory=16.00 GiB>> ```

Ah, indeed -- thanks @jhamman!

Closed via https://github.com/pydata/xarray/pull/7458

cc @dcherian

Looks like the latest distributed release is being used in the mypy CI build. https://github.com/dask/distributed/pull/7379 will be included in the next release which is currently slated for next Friday, December 16.

Not sure if it makes sense to temporarily use the main branch of distributed in CI until then, or use some other mypy-related workaround

It's not immediately clear to me why the docs build is failing. Has that been happening elsewhere?

@jrbourbeau is this something dask has thought about?

Thanks for the ping @Illviljan. Zero-padding dates did come up in the Dask calver discussion starting here https://github.com/dask/community/issues/100#issuecomment-704445214. In a nutshell, there was a slight preference towards using zero-padding (i.e. 2022.01.0 instead of 2022.1.0) because the calendar nature of the version is more explicit and string sorting and full-fledged package sorting (like one would do with packaging.version) give the same result.

As pointed out https://github.com/dask/community/issues/100#issuecomment-704468187 either convention is valid from a Python packaging perspective. FWIW I'm not aware of any issues that have come up from Dask using a zero-padded version number. The main thing that comes to mind is when checking out git tags for a specific release (e.g. git checkout 2021.04.0 and git checkout 2021.4.0 are not equivalent). That said, to my knowledge, this hasn't been an issue in practice.

EDIT: To be clear, I'm not advocating for one convention over the other -- just providing context around Dask's decision

https://github.com/dask/dask/pull/8125 was just merged and should resolve this issue

This should be closed via https://github.com/pydata/xarray/pull/5751. Again, thanks all for your efforts!

Thanks for hunting all this down @crusaderky, I'm currently grokking it. Also cc @hameerabbasi for visibility

Hrm, looking at the API docs for sparse.zeros_like, it appears to support shape=

```python In [1]: import sparse

In [2]: sparse.version Out[2]: '0.12.0'

In [3]: import numpy as np

In [4]: x = np.arange(6)

In [5]: y = sparse.zeros_like(x, shape=(3, 2))

In [6]: y Out[6]: <COO: shape=(3, 2), dtype=int64, nnz=0, fill_value=0>

In [7]: y.todense() Out[7]: array([[0, 0], [0, 0], [0, 0]]) ```

Also cc @crusaderky if you get a chance to look into this

Good point. CI is currently pulling in the latest sparse=0.12.0 release (which is also what I'm running locally).

It also appears there were some ordering concerns related to https://github.com/dask/dask/issues/5259 when test_chunk was added (xref https://github.com/pydata/xarray/pull/3202#discussion_r313098400)

Ah, that's a good point @keewis. Stepping back a bit, my current situation locally is I'm able to reproduce the xarray/tests/test_sparse.py::test_chunk failure with:

• the latest commit to dask (https://github.com/dask/dask/commit/76822dbc3da29767d7e6d70d73010046bd52604e)
• the latest commit to xarray (https://github.com/pydata/xarray/commit/e26aec9500e04f3b926b248988b976dbfcb9c632)
• numpy=1.21.1
• pandas=1.3.1

Rolling dask back to the 2021.7.2 release, but keeping everything else the same, test_chunk passes. After running git bisect between the latest dask commit and the 2021.7.2 release, https://github.com/dask/dask/pull/7939 is the point where test_chunk started failing (cc @jakirkham for visibility). I'll try to dig in a bit more to see if I can find the root cause of the failure

EDIT: I should note that the test_chunk failure has a similar traceback to https://github.com/dask/dask/issues/5259

Thanks for the ping @dcherian. Running against Dask's git history, https://github.com/dask/dask/pull/7939 is the place where xarray/tests/test_sparse.py::test_chunk started failing. We probably dropped some relevant compatibility code when we stopped supporting NumPy 1.17 and pandas 0.25.

Since xarray has a clearly defined policy for minimum dependencies, would dropping NumPy 1.17 and pandas 0.25 be an acceptable solution here?

Apologies for the breaking change. I'm happy to make sure we ping an Xarray dev next time we have a significant change to apply_gufunc to avoid surprises like this in the future.

Thanks for raising an issue @pont-us. I believe this may have been fixed by https://github.com/pydata/xarray/pull/5271 (which was merged just after the 0.18.0 release). Could you install the development version of xarray and try again to see if the problem persists?

Yeah, sorry about that. At least it was an easy fix : )

@dcherian could you try restarting CI to pull in the latest dask release? xref https://github.com/dask/community/issues/150#issuecomment-826025006

This should now be resolved upstream in Dask (xref https://github.com/dask/dask/pull/5399)