issues

Looking at an open_zarr computation from @rabernat I'm coming across intermediate values like the following:

```python

Future('zarr-adt-0f90b3f56f247f966e5ef01277f31374').result() ImplicitToExplicitIndexingAdapter(array=LazilyIndexedArray(array=<xarray.backends.zarr.ZarrArrayWrapper object at 0x7fa921fec278>, key=BasicIndexer((slice(None, None, None), slice(None, None, None), slice(None, None, None))))) ```

This object has many dependents, and so will presumably have to float around the network to all of the workers

```python

len(dep.dependents) 1781 ```

In principle this is fine, especially if this object is cheap to serialize, move, and deserialize. It does introduce a bit of friction though. I'm curious how hard it would be to build task graphs that generated these objects on the fly, or else removed them altogether. It is slightly more convenient from a task scheduling perspective for data access tasks to not have any dependencies.

This uses the automatic chunking in dask 0.18+ to chunk rasterio datasets in a nicely aligned way.

Currently this doesn't implement tests due to a difficulty in creating chunked tiff images.

This also uncovered some inefficiencies in how Dask doesn't align rechunking to existing chunk schemes.

• [x] Closes #2093
• [ ] Tests added (for all bug fixes or enhancements)
• [ ] Tests passed (for all non-documentation changes)
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I could use help on how the following:

• How to create tiled TIFF files in the tests
• The right way to merge different dtypes and block shapes in the tiff file. Currently I'm assuming that they're uniform

Given a tiled GeoTIFF image I'm looking for the best practice in reading it as a chunked dataset. I did this in this notebook by first opening the file with rasterio, looking at the block sizes, and then using those to inform the argument to chunks= in xarray.open_rasterio. This works, but is somewhat cumbersome because I also had to dive into the rasterio API. Do we want to provide defaults here?

In dask.array every time this has come up we've always shot it down, automatic chunking is error prone and hard to do well. However in these cases the object we're being given usually also conveys its chunking in a way that matches how dask.array thinks about it, so the extra cognitive load on the user has been somewhat low. Rasterio's model and API feel much more foreign to me though than a project like NetCDF or H5Py. I find myself wanting a chunks=True or chunks='100MB' option.

Thoughts on this? Is this in-scope? If so then what is the right API and what is the right policy for how to make xarray/dask.array chunks larger than GeoTIFF chunks?

Is anyone interested in submitting a tutorial to SciPy 2019? I think that it would be useful to have an official Xarray tutorial out there somewhere on the internet. This could be good motivation to create one.

https://www.scipy2019.scipy.org/tutorials

See also: https://github.com/pydata/xarray/issues/1882

I would like to call attention to https://github.com/JiaweiZhuang/xESMF/issues/3#issuecomment-354668897 . It seems like the xESMF package does regridding in a way that at least some XArray users find sensible. It should probably make use of, but does not currently use apply_ufunc, and is not particularly parallelizable (or at least that is my understanding). It could be that some modest development by someone more familiar with XArray could have a large impact by properly using apply_ufunc within that codebase.

I apologize for posting an issue about another package in this issue tracker. Feel free to close.

cc @JiaweiZhuang

I would like to have an XArray that has scipy.sparse arrays rather than numpy arrays. Is this in scope?

What would need to happen within XArray to support this?

Hi All,

There are a couple changes coming in dask that might affect XArray code:

1. We're replacing the get=dask.threaded.get keyword with scheduler='threads'
2. We're replacing dask.set_options(...) with dask.config.set(...)

Both of the old systems will still work, at least for a version or two, but we plan to remove them in the future. I thought I'd bring these changes up here so that we can plan a clean deprecation within XArray. These are also both not yet released, so both features are still up for discussion if this community has additional constraints.

Anecdotally I find that when I call .plot() on a dataset object that holds dask arrays compute gets called twice. Why is this? I'm curious if this is something that should be resolved.

Fixes https://github.com/dask/dask/issues/4291

• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

When working with @jhamman profiling the to_zarr method on large datasets I wanted the ability to run through the to_zarr setup code, but avoid waiting on the dask computation to finish. In many functions in Dask proper our to_foo methods have a compute=False keyword that returns a dask.delayed object on which people can call compute later if desired.

cc @jhamman @rabernat @jakirkham (who has been looking at similar questions within dask.array.Array.store)

When using open_zarr on a dataset with many variables the keynames include the variable name, like

('zarr-temperature-1234', 1, 3, 2)

In the distributed scheduler these keynames get shortened to prefixes like zarr-temperature and used both for scheduling heuristics (all keys with the same prefix are expected to take similar-ish amounts of time) and for diagnostics, such as in the progress bar plot below:

We may want to avoid including the variable name into the keyname here in order to avoid breaking these out into several groups. Instead you might consider putting the variable name within the key as another member of the tuple like the following:

('zarr-1234', 'temperature', 1, 3 ,2)

This internal state was changed in the latest release

• [x] Closes #1903 (remove if there is no corresponding issue, which should only be the case for minor changes)
• [ ] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

• [ ] Closes #xxxx
• [x] Tests added / passed
• [ ] Passes flake8 xarray
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

I had a free half hour so I decided to run autoflake and autopep8 tools on the codebase. flake8 xarray passes. I copied over the exclusions that we use within dask/distributed and extended the line length to 120. You may wish to review these decisions.

Code Sample, a copy-pastable example if possible

python ds = xarray.open_dataset(...) da = ds.variables['...'] da.<tab>

Problem description

This starts reading data. I don't know why. I'm using XArray against a FUSE system that is both expensive (it's targetting Google Cloud Storage) and also has logging. I can see that auto-completion immediately starts a lot of file reading on the file system.

Output of xr.show_versions()

Previously this erroneously pointed to an optimize function, likely a copy-paste error.

For testing this also redirects the .compute methods to use the dask.compute function directly if dask.version >= '0.16.0'.

Closes #1759

• [x] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [x] Tests added (for all bug fixes or enhancements)
• [x] Tests passed (for all non-documentation changes)
• [x] Passes git diff upstream/master **/*py | flake8 --diff (remove if you did not edit any Python files)
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

This integrates the new dask interface methods into XArray. This will place XArray as a first-class dask collection and help in particular with newer dask.distributed features.

• [x] Closes https://github.com/pangeo-data/pangeo/issues/5
• [x] Tests added / passed
• [x] Passes git diff upstream/master **/*py | flake8 --diff
• [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

Builds on work from @jcrist here: https://github.com/dask/dask/pull/2748 Depends on https://github.com/dask/dask/pull/2847

Some XArray loading functions provide keys that don't adhere to dask conventions used for naming.

We can solve this in XArray by using names like 'load-' + dask.base.tokenize(stuff) or in dask by trying to identify and avoid names like these. It might be wise to attempt both. I expect that this will be easier to solve in XArray (though that's also in my own self interest :))

Fixes https://github.com/pydata/xarray/issues/1344

• [x] closes #xxxx
• [x] tests added / passed
• [x] passes git diff upstream/master | flake8 --diff
• [x] whatsnew entry (not sure what to do here, is there a new section? It looks like the last release was yesterday)

It would be convenient to load constituent dask.arrays into memory as dask.arrays rather than as numpy arrays. This would help with distributed computations where we want to load a large amount of data into distributed memory once and then iterate on the full xarray dataset repeatedly without reloading from disk every time.

We can probably solve this from either side:

1. XArray could make a .persist method that replaced all of its dask.arrays with a persisted version of that array

```python import dask

dset.x, dset.y, dset.z = dask.persist(dset.x, dset.y, dset.z) ```

1. We could look into the Dask duck type solution again https://github.com/dask/dask/pull/1068

cc @shoyer @jcrist @rabernat @pwolfram

This would avoid a mismatch between clients and workers when using dask.distributed

python In [2]: os.path.abspath('my-local-path') Out[2]: '/home/mrocklin/my-local-path'