issues

What happened?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

• [ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [ ] Complete example — the example is self-contained, including all data and the text of any traceback.
• [ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [ ] New issue — a search of GitHub Issues suggests this is not a duplicate.
• [ ] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

No response

Anything else we need to know?

No response

Environment

Is your feature request related to a problem?

I'm playing around with xarray performance and would like a way to easily tweak chunk sizes. I'm able to do this by backing out what xarray chooses in an open_zarr call and then provide the right chunks= argument. I'll admit though that I wouldn't mind giving Xarray a value like "1 GiB" though and having it use that when determining "auto" chunk sizes.

Dask array does this in two ways. We can provide a value in chunks as like the following:

python x = da.random.random(..., chunks="1 GiB")

We also refer to a value in Dask config

```python In [1]: import dask

In [2]: dask.config.get("array.chunk-size") Out[2]: '128MiB' ```

This is not very important (I'm unblocked) but I thought I'd mention it in case someone is looking for some fun work 🙂

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

I'm looking at a pangeo gallery workflow that suffers from poor load balancing because objects of type xarray.core.indexing.ImplicitToExplicitIndexingAdapter are being interpretted as 48B when in fact, I suspect, they are significantly larger to move around.

I'm seeing number of processing tasks charts that look like the following, which is a common sign of the load balancer not making good decisions, which is most commonly caused by poor data size measurements:

tl;dr: Can XArray enable user-defined logical dtypes on top of physical NumPy arrays ?

The Need for New Datatypes

NumPy's dtypes (int, float, etc.) are appropriate for many, but not all cases. There are a variety of situations where we want numpy-like array semantics (broadcasting, memory layout) but with different element properties. Use cases include the following:

1. Datetimes with timezones
2. Categorical values (such as for land-use in climate data)
3. IPv4 or IPv6 addresses
4. ...

Currently dtypes need to be added directly to the NumPy source code. This is a high barrier for many community members, requires general approval (there can be only one datetime implementation) (this is good and bad), and limits experimentation. There is value to supporting user-definable datatypes.

This is hard to do in NumPy

Ideally we would implement extensible user-defined dtypes within NumPy (and there may be long-standing plans to do just this). However, changing NumPy today is hard, both because it's hard to find developers who are comfortable operating at that level and because the backwards compatibility pressure on NumPy is large.

So as an alternative, we might consider lightly wrapping NumPy arrays in a new object that also includes extra dtype information. For example we might wrap an int64 numpy array with some datetime/timezone metadata to achieve a logical datetime array using a physical int64 array. We continue using NumPy as is but use this higher layer when necessary for more complex dtypes.

However "lightly wrapping" NumPy arrays is hard to do while still maintaining a closed system where all operations remain consistent (raw NumPy arrays inevitably leak through). Additionally, asking communities to switch to new libraries is socially quite challenging.

XArray is well placed

Fortunately XArray appears to have already solved some of these technical and social challenges. XArray lightly wraps NumPy arrays in a consistent manner. NumPy-like operations on XArrays remain XArrays. Interactions with other NumPy arrays are well defined. XArray has also attracted an active user/developer community and has attained general respect from the broader ecosystem. XArray seems to be hackable, benefits from a decently active community, and is not yet under as much backwards compatibility pressure.

So question: Is it sensible to add logical dtype information to XArray? Can this be done with only moderate effort and maintenance costs to the XArray project? If the answer is "yes, probably", then what is the right way to go about this?

Code Sample, a copy-pastable example if possible

python from xarray.core.indexing import LazilyOuterIndexedArray import numpy as np x = LazilyOuterIndexedArray(np.ones((5, 5))) x[:3]

```python-traceback

AttributeError Traceback (most recent call last) <ipython-input-4-42bee9beb30a> in <module> ----> 1 x[:3]

~/workspace/xarray/xarray/core/indexing.py in getitem(self, indexer) 518 array = LazilyVectorizedIndexedArray(self.array, self.key) 519 return array[indexer] --> 520 return type(self)(self.array, self._updated_key(indexer)) 521 522 def setitem(self, key, value):

~/workspace/xarray/xarray/core/indexing.py in _updated_key(self, new_key) 483 484 def _updated_key(self, new_key): --> 485 iter_new_key = iter(expanded_indexer(new_key.tuple, self.ndim)) 486 full_key = [] 487 for size, k in zip(self.array.shape, self.key.tuple):

AttributeError: 'slice' object has no attribute 'tuple' ```

Problem description

Dask array meta computations like to run x[:0, :0] on input arrays. This breaks with this class.

This is on master