issues: 732378121

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
732378121 MDExOlB1bGxSZXF1ZXN0NTEyMzI4MDY4 4550 WIP: Zarr chunks refactor 35919497 closed 0     1 2020-10-29T14:44:31Z 2020-12-10T10:28:06Z 2020-11-10T16:08:53Z COLLABORATOR   0 pydata/xarray/pulls/4550

This work aims to harmonize the way zarr deals with chunking to have similar behavior for all other backends and unify the code. Most of the changes involve the new API, apiv2.py, except for some changes in the code that has been added with the merge of https://github.com/pydata/xarray/pull/4187.

main changes: - refactor apiv2.dataset_from_backend_dataset function. - move get_chunks from zarr to dataset.

current status: - in apiv2.open_dataset chunks='auto' and chunks={} now has the same beahviuor - in apiv2.open_dataset for all the backends now the default chunking is provided by the backend, if it is not available it uses one big chunk.

Missing points: - standardize the key in encodings to define the on-disk chunks: chunksizes - add a specific key in encodings for preferred chunking (currently it is used chunks)

There is one open point to be discussed yet: dataset.chunks and open_dataset(..., chunks=...) have different behaviors. dataset.chunks(chunks={}) opens the dataset with only one chunk per variable, while in open_dataset(..., chunks={}) it uses encodings['chunks'], when available.

Note that also chunks=None has a different behaviour: open_dataset(..., chunks=None) (or open_dataset(...), it's the deafult) returns variables without chunks, while dataset.chunk(chunks=None) (or dataset.chunk(), it's the default) has the same behavior of dataset.chunk(chunks=None). Probably it's not worth changing it.

  • [x] related to https://github.com/pydata/xarray/issues/4496
  • [ ] Tests added
  • [x] Passes isort . && black . && mypy . && flake8
  • [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
  • [ ] New functions/methods are listed in api.rst
 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4550/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     13221727 pull

Links from other tables