issue_comments

@lskopintseva - This feature has not been implemented in Xarray (yet). In the meantime, you might find something like this helpful:

python ds = xr.open_dataset("dataset.nc") for v in ds.data_vars: # get variable chunksizes chunksizes = ds[v].encoding.get('chunksizes', None) if chunksizes is not None: chunks = dict(zip(ds[v].dims, chunksizes)) ds[v] = ds[v].chunk(chunks) # chunk the array using the underlying chunksizes

FWIW, I think this would be a nice feature to add to the netcdf4 and h5netcdf backends in Xarray. Contributions welcome!

It seems to me that the there are lots of "layers" of "chunking", especially when you are talking about chunking an entire dataset,

To simplify a little bit, here we are only talking about reading a single store, i.e. one netcdf file or one zarr group. Also out of scope is the underlying storage medium (e.g. block size).

Can we overload the chunks argument in open_xxx to do this? We are already adding support for chunks="auto" ...

This gets tricky, because we may want slightly different behavior depending on whether the underlying array store is chunked.

should we have an option like chunk_size='native', or chunk_size='100MB'

Can we overload the chunks argument in open_xxx to do this? We are already adding support for chunks="auto" ...

We discussed this issue today in our pangeo coffee break. We think the following plan would be good:

• [ ] Write a function called auto_chunk(variable) which examines a variable for the presence of a chunks attribute in encoding or within the data itself. Returns a new variable with chunked data.
• [ ] Refactor open_zarr to call this function
• [ ] Add it also to open_dataset to enable auto-chunking of netCDF and geotiff data

Should we have an option like chunk_size='native', or chunk_size='100MB', with chunks chosen to align with source chunks.

cc @kmpaul who wanted to review this conversation.

@Zac-HD - We merged #1457 yesterday which should give us a platform to test any improvements we make related to this issue.

@Zac-HD - thanks for you detailed report.

ping me again when you get started on some benchmarking and feel free to chime in further to #1457.

No block should include data from multiple files (near-absolute, due to locking - though concurrent read is supported on lower levels?)

Hopefully we can find some optimizations that help with this. I routinely want to do this, though I understand why its not always a good idea.

@Zac-HD - I'm about to put up a PR with some initial benchmarking functionality (#1457). Are you open to putting together PR for the features you've described above? Hopefully, these two can work together.

As for the API changes related to this issue, I'd propose the following:

Use the chunks keyword to support 3 additional options

python def open_dataset(filename_or_obj, ..., chunks=None, ...): """Load and decode a dataset from a file or file-like object. Parameters ---------- .... chunks : int or dict or set or 'auto' or 'disk', optional If chunks is provided, it used to load the new dataset into dask arrays. ``chunks={}`` loads the dataset with dask using a single chunk for all arrays. ... """

• int: chunk each dimension by chunks
• dict: Dictionary with keys given by dimension names and values given by chunk sizes. In general, these should divide the dimensions of each dataset
• set (or list or tuple) of str: chunk the dimension(s) provided by some heuristic, try to keep the chunk shape/size compatible with the storage of the data on disk and for use with dask
• 'auto' (str): chunk the array(s) using some auto-magical heuristic that is compatible with the storage of the data on disk and is semi-optimized (in size) for use with dask
• 'disk' (str): use the chunksize of the netCDF variable directly.

I think its unavoidable that users understand how their data will be processed (e.g., whether operations will be mapped over time or space). But maybe some sort of heuristics (if not a fully automated solution) are possible.

For example, maybe chunks={'time'} (note the set rather than a dict) could indicate "divide me into automatically chosen chunks over the time dimension". It's still explicit about how chunking is being done, but comes closer to expressing the intent rather than the details.

I'd certainly support a warning when dask chunks do not align with the on-disk chunks.

Beyond that, I think we could work on a utility for automatically determining chunks sizes for xarray using some heuristics. Before we go there though, I think we really should develop some performance benchmarks. We're starting to get a lot of questions/issues about performance and it seems like we need some benchmarking to happen before we can really start fixing the underlying issues.

My main concern is that netCDF4 chunk sizes (e.g., ~10-100KB in that blog post) are often much smaller than well sized dask chunks (10-100MB, per the Dask FAQ).

I do think it would be appropriate to issue a warning if you are making dask chunks that don't line up nicely with chunks on disk to avoid performance issues (in general each chunk on disk should usually end up on only one chunk in dask), but there are lots of options for aggregating to larger chunks and it's hard to choose the best way to do that without knowing how the data will be used.