issues

• [x] Closes https://github.com/pydata/xarray/issues/8423
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

cc @dcherian as you might find this interesting

Is your feature request related to a problem?

Currently the h5netcdf engine supports opening remote files, but only already open file-like objects (e.g. s3fs.open(...)), not string paths like s3://.... There are situations where I'd like to use string paths instead of open file-like objets

• Opening files can sometimes be slow (xref https://github.com/fsspec/s3fs/issues/816)
• When using parallel=True for opening lots of files, serializing open file-like objects back and forth from a remote cluster can be slow
• Some systems (e.g. NASA Earthdata) only hand out credentials that are valid when run in the same region as the data. Being able to use parallel=True + storage_options would be convenient/performant in that case.

Describe the solution you'd like

It would be nice if I could do something like the following:

python ds = xr.open_mfdataset( files, # A bunch of files like `s3://bucket/file` engine="h5netcdf", ... parallel=True, storage_options={...}, # fsspec-compatible options )

and have my files opened prior to handing off to h5netcdf. storage_options is already supported for Zarr, so hopefully extending to h5netcdf feels natural.

Describe alternatives you've considered

No response

Additional context

No response

I'm not totally sure why single-threaded was being set to begin with, so I might be missing some important context here

Closes https://github.com/pydata/xarray/issues/7483

xref https://github.com/PyCQA/isort/issues/2077

Closes https://github.com/pydata/xarray/issues/7490

What happened?

When I attempt to run pre-commit locally I get the following error:

``` ❯ pre-commit run --all-files [INFO] Installing environment for https://github.com/PyCQA/isort. [INFO] Once installed this environment will be reused. [INFO] This may take a few minutes... An unexpected error has occurred: CalledProcessError: command: ('/Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/bin/python', '-mpip', 'install', '.') return code: 1 expected return code: 0 stdout: Processing /Users/james/.cache/pre-commit/repo_eax0t7q Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing wheel metadata: started Preparing wheel metadata: finished with status 'error'

stderr: DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. ERROR: Command errored out with exit status 1: command: /Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/bin/python /Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/lib/python3.9/site-packages/pip/vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/k/lx1rdvqn253gd1wrcx__5frm0000gn/T/tmphq1i398_ cwd: /private/var/folders/k_/lx1rdvqn253gd1wrcx__5frm0000gn/T/pip-req-build-4ywk4y5l Complete output (14 lines): Traceback (most recent call last): File "/Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/lib/python3.9/site-packages/pip/vendor/pep517/in_process/_in_process.py", line 280, in <module> main() File "/Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 263, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 133, in prepare_metadata_for_build_wheel return hook(metadata_directory, config_settings) File "/private/var/folders/k/lx1rdvqn253gd1wrcx__5frm0000gn/T/pip-build-env-bvdexj4q/overlay/lib/python3.9/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False) File "/private/var/folders/k_/lx1rdvqn253gd1wrcx__5frm0000gn/T/pip-build-env-bvdexj4q/overlay/lib/python3.9/site-packages/poetry/core/factory.py", line 57, in create_poetry raise RuntimeError("The Poetry configuration is invalid:\n" + message) RuntimeError: The Poetry configuration is invalid: - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

    ----------------------------------------
WARNING: Discarding file:///Users/james/.cache/pre-commit/repo_eax0t7q. Command errored out with exit status 1: /Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/bin/python /Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/k_/lx1rdvqn253gd1wrcx__5frm0000gn/T/tmphq1i398_ Check the logs for full command output.
ERROR: Command errored out with exit status 1: /Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/bin/python /Users/james/.cache/pre-commit/repo_eax0t7q/py_env-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/k_/lx1rdvqn253gd1wrcx__5frm0000gn/T/tmphq1i398_ Check the logs for full command output.

Check the log at /Users/james/.cache/pre-commit/pre-commit.log ```

It also looks like we're seeing the same thing in CI https://results.pre-commit.ci/run/github/13221727/1675098797.6-zAnTMbQXC74StmKz5PFw

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Noticed a tiny typo when reading through https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html

• [x] Closes https://github.com/dask/distributed/issues/6842
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

This is a quickfix similar to https://github.com/dask/dask/pull/9337. Note there's a conversation happening over in https://github.com/dask/distributed/issues/6806 around what testing utilities in distributed should be considered public.

Just wanting to check the test suite before issuing a dask release. Apologies for the noise.

xarray/tests/test_sparse.py::test_chunk now passes with the latest sparse=0.13.0 release

• [x] Closes https://github.com/pydata/xarray/issues/5654
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

In https://github.com/dask/distributed/pull/5170, @fjetter pointed out that distributed was incorrectly using force_to_top in our isort configuration (we should have been using float_to_top instead). Just a minor point, but since xarrays isort configuration influenced what we did over in dask and distributed, I thought it was worth letting the Xarray devs know about it

https://github.com/pydata/xarray/blob/8f5b4a185b304ab68723c390b1ad88e57f9a60d6/setup.cfg#L164

This PR updates how we construct the HighLevelGraph in subset_dataset_to_block. Currently we create a HighLevelGraph and then manually add a few new layers, which are dicts, directly to hlg.layers.

https://github.com/pydata/xarray/blob/eec2b59bd08498fe3d50d8549b540d0a8409c7ac/xarray/core/parallel.py#L537-L546

However since (in more recent versions of Dask) HighLevelGraph layers are expected to be Layer class instances this can result in unexpected errors (xref https://github.com/pydata/xarray/issues/5077#issuecomment-832359943). Instead of manually adding new layers to hlg.layers, this PR proposes we create a new HighLevelGraph altogether to ensure that hlg.layers won't contain any raw dict layers.