issues

What happened?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

I'm playing around with xarray performance and would like a way to easily tweak chunk sizes. I'm able to do this by backing out what xarray chooses in an open_zarr call and then provide the right chunks= argument. I'll admit though that I wouldn't mind giving Xarray a value like "1 GiB" though and having it use that when determining "auto" chunk sizes.

Dask array does this in two ways. We can provide a value in chunks as like the following:

python x = da.random.random(..., chunks="1 GiB")

We also refer to a value in Dask config

```python In [1]: import dask

In [2]: dask.config.get("array.chunk-size") Out[2]: '128MiB' ```

This is not very important (I'm unblocked) but I thought I'd mention it in case someone is looking for some fun work 🙂

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Looking at an open_zarr computation from @rabernat I'm coming across intermediate values like the following:

```python

Future('zarr-adt-0f90b3f56f247f966e5ef01277f31374').result() ImplicitToExplicitIndexingAdapter(array=LazilyIndexedArray(array=<xarray.backends.zarr.ZarrArrayWrapper object at 0x7fa921fec278>, key=BasicIndexer((slice(None, None, None), slice(None, None, None), slice(None, None, None))))) ```

This object has many dependents, and so will presumably have to float around the network to all of the workers

```python

len(dep.dependents) 1781 ```

In principle this is fine, especially if this object is cheap to serialize, move, and deserialize. It does introduce a bit of friction though. I'm curious how hard it would be to build task graphs that generated these objects on the fly, or else removed them altogether. It is slightly more convenient from a task scheduling perspective for data access tasks to not have any dependencies.

I'm looking at a pangeo gallery workflow that suffers from poor load balancing because objects of type xarray.core.indexing.ImplicitToExplicitIndexingAdapter are being interpretted as 48B when in fact, I suspect, they are significantly larger to move around.

I'm seeing number of processing tasks charts that look like the following, which is a common sign of the load balancer not making good decisions, which is most commonly caused by poor data size measurements:

tl;dr: Can XArray enable user-defined logical dtypes on top of physical NumPy arrays ?

The Need for New Datatypes

NumPy's dtypes (int, float, etc.) are appropriate for many, but not all cases. There are a variety of situations where we want numpy-like array semantics (broadcasting, memory layout) but with different element properties. Use cases include the following:

1. Datetimes with timezones
2. Categorical values (such as for land-use in climate data)
3. IPv4 or IPv6 addresses
4. ...

Currently dtypes need to be added directly to the NumPy source code. This is a high barrier for many community members, requires general approval (there can be only one datetime implementation) (this is good and bad), and limits experimentation. There is value to supporting user-definable datatypes.

This is hard to do in NumPy

Ideally we would implement extensible user-defined dtypes within NumPy (and there may be long-standing plans to do just this). However, changing NumPy today is hard, both because it's hard to find developers who are comfortable operating at that level and because the backwards compatibility pressure on NumPy is large.

So as an alternative, we might consider lightly wrapping NumPy arrays in a new object that also includes extra dtype information. For example we might wrap an int64 numpy array with some datetime/timezone metadata to achieve a logical datetime array using a physical int64 array. We continue using NumPy as is but use this higher layer when necessary for more complex dtypes.

However "lightly wrapping" NumPy arrays is hard to do while still maintaining a closed system where all operations remain consistent (raw NumPy arrays inevitably leak through). Additionally, asking communities to switch to new libraries is socially quite challenging.

XArray is well placed

Fortunately XArray appears to have already solved some of these technical and social challenges. XArray lightly wraps NumPy arrays in a consistent manner. NumPy-like operations on XArrays remain XArrays. Interactions with other NumPy arrays are well defined. XArray has also attracted an active user/developer community and has attained general respect from the broader ecosystem. XArray seems to be hackable, benefits from a decently active community, and is not yet under as much backwards compatibility pressure.

So question: Is it sensible to add logical dtype information to XArray? Can this be done with only moderate effort and maintenance costs to the XArray project? If the answer is "yes, probably", then what is the right way to go about this?

Given a tiled GeoTIFF image I'm looking for the best practice in reading it as a chunked dataset. I did this in this notebook by first opening the file with rasterio, looking at the block sizes, and then using those to inform the argument to chunks= in xarray.open_rasterio. This works, but is somewhat cumbersome because I also had to dive into the rasterio API. Do we want to provide defaults here?

In dask.array every time this has come up we've always shot it down, automatic chunking is error prone and hard to do well. However in these cases the object we're being given usually also conveys its chunking in a way that matches how dask.array thinks about it, so the extra cognitive load on the user has been somewhat low. Rasterio's model and API feel much more foreign to me though than a project like NetCDF or H5Py. I find myself wanting a chunks=True or chunks='100MB' option.

Thoughts on this? Is this in-scope? If so then what is the right API and what is the right policy for how to make xarray/dask.array chunks larger than GeoTIFF chunks?

Is anyone interested in submitting a tutorial to SciPy 2019? I think that it would be useful to have an official Xarray tutorial out there somewhere on the internet. This could be good motivation to create one.

https://www.scipy2019.scipy.org/tutorials

See also: https://github.com/pydata/xarray/issues/1882

I would like to call attention to https://github.com/JiaweiZhuang/xESMF/issues/3#issuecomment-354668897 . It seems like the xESMF package does regridding in a way that at least some XArray users find sensible. It should probably make use of, but does not currently use apply_ufunc, and is not particularly parallelizable (or at least that is my understanding). It could be that some modest development by someone more familiar with XArray could have a large impact by properly using apply_ufunc within that codebase.

I apologize for posting an issue about another package in this issue tracker. Feel free to close.

cc @JiaweiZhuang

I would like to have an XArray that has scipy.sparse arrays rather than numpy arrays. Is this in scope?

What would need to happen within XArray to support this?

Hi All,

There are a couple changes coming in dask that might affect XArray code:

1. We're replacing the get=dask.threaded.get keyword with scheduler='threads'
2. We're replacing dask.set_options(...) with dask.config.set(...)

Both of the old systems will still work, at least for a version or two, but we plan to remove them in the future. I thought I'd bring these changes up here so that we can plan a clean deprecation within XArray. These are also both not yet released, so both features are still up for discussion if this community has additional constraints.

Anecdotally I find that when I call .plot() on a dataset object that holds dask arrays compute gets called twice. Why is this? I'm curious if this is something that should be resolved.

Code Sample, a copy-pastable example if possible

python from xarray.core.indexing import LazilyOuterIndexedArray import numpy as np x = LazilyOuterIndexedArray(np.ones((5, 5))) x[:3]

```python-traceback

AttributeError Traceback (most recent call last) <ipython-input-4-42bee9beb30a> in <module> ----> 1 x[:3]

~/workspace/xarray/xarray/core/indexing.py in getitem(self, indexer) 518 array = LazilyVectorizedIndexedArray(self.array, self.key) 519 return array[indexer] --> 520 return type(self)(self.array, self._updated_key(indexer)) 521 522 def setitem(self, key, value):

~/workspace/xarray/xarray/core/indexing.py in _updated_key(self, new_key) 483 484 def _updated_key(self, new_key): --> 485 iter_new_key = iter(expanded_indexer(new_key.tuple, self.ndim)) 486 full_key = [] 487 for size, k in zip(self.array.shape, self.key.tuple):

AttributeError: 'slice' object has no attribute 'tuple' ```

Problem description

Dask array meta computations like to run x[:0, :0] on input arrays. This breaks with this class.

This is on master

When working with @jhamman profiling the to_zarr method on large datasets I wanted the ability to run through the to_zarr setup code, but avoid waiting on the dask computation to finish. In many functions in Dask proper our to_foo methods have a compute=False keyword that returns a dask.delayed object on which people can call compute later if desired.

cc @jhamman @rabernat @jakirkham (who has been looking at similar questions within dask.array.Array.store)

When using open_zarr on a dataset with many variables the keynames include the variable name, like

('zarr-temperature-1234', 1, 3, 2)

In the distributed scheduler these keynames get shortened to prefixes like zarr-temperature and used both for scheduling heuristics (all keys with the same prefix are expected to take similar-ish amounts of time) and for diagnostics, such as in the progress bar plot below:

We may want to avoid including the variable name into the keyname here in order to avoid breaking these out into several groups. Instead you might consider putting the variable name within the key as another member of the tuple like the following:

('zarr-1234', 'temperature', 1, 3 ,2)

Code Sample, a copy-pastable example if possible

python ds = xarray.open_dataset(...) da = ds.variables['...'] da.<tab>

Problem description

This starts reading data. I don't know why. I'm using XArray against a FUSE system that is both expensive (it's targetting Google Cloud Storage) and also has logging. I can see that auto-completion immediately starts a lot of file reading on the file system.

Output of xr.show_versions()

Some XArray loading functions provide keys that don't adhere to dask conventions used for naming.

We can solve this in XArray by using names like 'load-' + dask.base.tokenize(stuff) or in dask by trying to identify and avoid names like these. It might be wise to attempt both. I expect that this will be easier to solve in XArray (though that's also in my own self interest :))

It would be convenient to load constituent dask.arrays into memory as dask.arrays rather than as numpy arrays. This would help with distributed computations where we want to load a large amount of data into distributed memory once and then iterate on the full xarray dataset repeatedly without reloading from disk every time.

We can probably solve this from either side:

1. XArray could make a .persist method that replaced all of its dask.arrays with a persisted version of that array

```python import dask

dset.x, dset.y, dset.z = dask.persist(dset.x, dset.y, dset.z) ```

1. We could look into the Dask duck type solution again https://github.com/dask/dask/pull/1068

cc @shoyer @jcrist @rabernat @pwolfram

This would avoid a mismatch between clients and workers when using dask.distributed

python In [2]: os.path.abspath('my-local-path') Out[2]: '/home/mrocklin/my-local-path'