issues

This issue proposes exposing Xarray's Variable class as a stand-alone array class with named axes (dims) and arbitrary metadata (attrs) but without coordinates (indexes). Yes, this already exists but the Variable class in currently inseparable from our Pandas dependency, despite not utilizing any of its functionality. What would this entail?

The biggest change would be in making Pandas an optional dependency and isolating any imports. This change could be confined to the Variable object or could be propagated further as the Explicit Indexes work proceeds (#1603).

Why?

Within Xarray, the Variable class is a vital building block for many of our internal data structures. Recently, the utility of a simple array with named dimensions has been highlighted by a few potential user communities:

• Scikit-learn: https://github.com/scikit-learn/enhancement_proposals/pull/18
• PyTorch: (https://pytorch.org/tutorials/intermediate/named_tensor_tutorial.html, http://nlp.seas.harvard.edu/NamedTensor)

An example from the above linked SLEP as to why users may not want Pandas a dependency in Xarray:

@amueller: ...If we go this route, I think we need to make xarray, and therefore pandas, a mandatory dependency... ... @adrinjalali: ...And we still do have the option of making a NamedArray. xarray uses the pandas' index classes for the indexing and stuff, which is something we really don't need...

Since we already have a class developed that meets these applications' use cases, its seems only prudent to evaluate the feasibility in exposing the Variable as a low-level api object.

In conclusion, I'm not sure this is currently worth the effort but its probably worth exploring at this point.

Hello Xarray Community and @pydata/xarray,

Starting next week, we will be hosting a bi-weekly 30-minute community/developers meeting. The goal of this meeting is to help coordinate Xarray development efforts and better connect the user/developer community.

When

Every other Wednesday at 8:30a PT (11:30a ET) beginning April 29th, 2020.

Calendar options: - Google Calendar - Ical format

Where

https://us02web.zoom.us/j/87503265754?pwd=cEFJMzFqdTFaS3BMdkx4UkNZRk1QZz09

Rolling agenda and meeting notes

We'll keep a rolling agenda and set of meeting notes - Through Sept. 2022. - Starting October 2022 (requires sign-in) - Starting March 2024

What is your issue?

We have discussed many time deprecating xarray.open_zarr in favor of xarray.open_dataset(..., engine='zarr'). This issue tracks that process and is a place for us to discuss any issues that may arise as a result of the change.

xref: https://github.com/pydata/xarray/issues/2812, https://github.com/pydata/xarray/issues/7293

This PR deprecates open_zarr in favor of open_dataset(..., engine='zarr').

• [x] Closes #7495
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

What is your issue?

We have recently had a few reports expressing frustration with our minimum dependency version policy. This issue aims to discuss if changes to our policy are needed.

Background

1. Our current minimum dependency versions policy reads:

   Minimum dependency versions

   Xarray adopts a rolling policy regarding the minimum supported version of its dependencies:

   • Python: 24 months (NEP-29)
   • numpy: 18 months (NEP-29)
   • all other libraries: 12 months

   This means the latest minor (X.Y) version from N months prior. Patch versions (x.y.Z) are not pinned, and only the latest available at the moment of publishing the xarray release is guaranteed to work.

   You can see the actual minimum tested versions:

   pydata/xarray

2. We have a script that checks versions and dates and advises us on when to bump minimum versions.

   https://github.com/pydata/xarray/blob/main/ci/min_deps_check.py

Diagnosis

1. Our policy and min_deps_check.py script have greatly reduced our deliberations on which versions to support and the maintenance burden of supporting out dated versions of dependencies.
2. We likely need to update our policy and min_deps_check.py script to properly account for Python's SEMVER bugfix releases. Depending on how you interpret the policy, we may have prematurely dropped Python 3.8 (see below for a potential action item).

Discussion questions

1. Is the policy working as designed, are the support windows documented above still appropriate for where Xarray is at today?
2. Is this policy still in line with how our peer libraries are operating?

Action items

1. There is likely a bug in the patch-version comparison in the minimum Python version. Moreover, we don't differentiate between bugfix and security releases. I suggest we have a special policy for our minimum supported Python version that reads something like: > Python: 24 months from the last bugfix release (security releases are not considered).

xref: https://github.com/pydata/xarray/issues/4179, https://github.com/pydata/xarray/pull/7461

Moderators note: I suspect a number of folks will want to comment on this issue with "Please support Python 3.8 for longer...". If that is the nature of your comment, please just give this a ❤️ reaction rather than filling up the discussion.

There have been a number of threads discussing possible improvements/extensions to xray indexing. The current indexing behavior for isel is orthogonal indexing - in other words, each coordinate is treated independently (see #214 and #411 for more discussion).

So the question: what is the best way to incorporate diagonal or pointwise indexing in xray? I see two main goals / applications: 1. support simple form of numpy style integer array indexing 2. support pointwise array indexing along coordinates via computation of nearest-neighbor indexes - I think this can also be thought of as a form of resampling.

Input from @WeatherGod, @wholmgren, and @shoyer would be great.

Is your feature request related to a problem?

Most users of Xarray/Pandas start with an IO call of some sort. In Xarray, our open_dataset(..., engine=engine) interface provides an extensible interface to more complex backends (NetCDF, Zarr, GRIB, etc.). For tabular data types, we have traditionally pointed users to Pandas. While this works for users that are comfortable with Pandas, it is an added hurdle to users getting started with Xarray.

Describe the solution you'd like

It should be easy and obvious how a user can get a CSV (or other tabular data) into Xarray. Ideally, we don't force the user to use a third part library.

Describe alternatives you've considered

I can think of three possible solutions:

1. We expose a new function read_csv, it may do something like this:

python def read_csv(filepath_or_buffer, **kwargs): df = pd.read_csv(filepath_or_buffer, **kwargs) ds = xr.Dataset.from_dataframe(df) return ds

1. We develop a storage backend to support reading CSV-like data:

python ds = open_dataset(filepath, engine='csv')

1. We copy (1) as an example and put it in Xarray's documentation. Explicitly showing how you would use Pandas to produce a Dataset from a CSV.

Is your feature request related to a problem?

I recently learned about a meta-tensor library called Aesara which got me wondering if it would be a good array backend for Xarray.

Aesara is a Python library that allows you to define, optimize/rewrite, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It is composed of different parts: - Symbolic representation of mathematical operations on arrays - Speed and stability optimization - Efficient symbolic differentiation - Powerful rewrite system to programmatically modify your models - Extendable backends. Aesara currently compiles to C, Jax and Numba.

xref: https://github.com/aesara-devs/aesara/issues/352, @OriolAbril, @twiecki

Has anyone looked into this yet?

As was discussed in #608, we should honor the shape of the DataArray when selecting plot methods. Currently, we're squeezing the DataArray before plotting. This ends up plotting a line plot for a DataArray with shape (N, 1). We should find a way to plot a pcolormesh or imshow plot in this case. The trick will be figuring out what to do in _infer_interval_breaks.

• [x] Toward #6323
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [x] New functions/methods are listed in api.rst

What happened:

I'm getting a maximum recursion error when using the Pydap backend with Dask distributed. It seems the we're failing to successfully pickle the pydap backend store.

What you expected to happen:

Successful parallel loading of opendap dataset.

Minimal Complete Verifiable Example:

```python import xarray as xr from dask.distributed import Client

client = Client()

ds = xr.open_dataset('http://thredds.northwestknowledge.net:8080/thredds/dodsC/agg_terraclimate_pet_1958_CurrentYear_GLOBE.nc', engine='pydap', chunks={'lat': 1024, 'lon': 1024, 'time': 12}).load() ```

yields:

Killed worker on the client:

and the above mentioned recursion error on the workers:

Anything else we need to know?:

I've found this to be reproducible with a few kinds of Dask clusters. Setting Client(processes=False) does correct the problem at the expense of multiprocessiing.

Environment:

This applies to single plots and FacetGrids.

The current plotting behavior when using a projection that changes the plot aspect is as follows:

``` Python from xray.tutorial import load_dataset

ds = load_dataset('air_temperature')

ax = plt.subplot(projection=ccrs.LambertConformal()) ds.air.isel(time=0).plot(transform=ccrs.PlateCarree()) ax.coastlines() ax.gridlines() ```

Python fg = ds.air.isel(time=slice(0, 9)).plot(col='time', col_wrap=3, transform=ccrs.PlateCarree(), subplot_kws=dict(projection=ccrs.LambertConformal())) for ax in fg.axes.flat: ax.coastlines() ax.gridlines()

There are two problems here, I think both are related to the aspect of the subplot: 1. In the single case, the subplot aspect is correct but the colorbar is not scaled appropriately 2. In the FacetGrid case, the subplot aspects are not correct but the colorbar is.

Based on discussion in #781 and #782, it seems like a bad idea to infer (guess) the spacing of coordinates when they are unevenly spaced. As @ocefpaf points out:

guessing should be an active user choice, not the automatic behavior.

So the options moving forward are to 1. never infer the interval breaks and be okay with pcolormesh and imshow producing dissimilar plots, or 2. only infer the interval breaks when the coordinates are evenly spaced.

cc @clarkfitzg

It has come time to formalize the API for Xarray backends. We now have the following backends implemented in xarray:

| Backend | Read | Write | |----------------|------|-------| | netcdf4-python | x | x | | h5netcdf | x | x | | pydap | x | | | pynio | x | | | scipy | x | x | | rasterio* | x | | | zarr | x | x |

* currently does not inherit from backends.AbstractDatastore

And there are conversations about adding additional backends, for example:

• TileDB: https://github.com/pangeo-data/storage-benchmarks/issues/6
• PseudoNetCDF: #1905

However, as anyone who has worked on implementing or optimizing any of our current backends can attest, the existing DataStore API is not particularly user/developer friendly. @shoyer asked me to open an issue to discuss what a more user friendly backend API would look like so that is what this issue will be. I have left out a thorough description of the current API because, well, I don't think it can done in a succinct manner (thats the problem).

Note that @shoyer started down a API refactor some time ago in #1087 but that effort has stalled, presumably because we don't have a well defined set of development goals here.

cc @pydata/xarray

We need to come up with some clear rules for when and how xarray should propagate metadata (attrs/encoding). This has come up routinely (e.g. #25, #138, #442, #688, #828, #988, #1009, #1271, #1297, #1586) and we don't have a clear direction as to when to keep/drop metadata.

I'll take a first cut:

| operation | attrs | encoding | status | |------------ |------------ |------------ |------------ | | reduce | drop | drop | | | arithmetic | drop | drop | implemented | | copy | keep | keep | | | concat | keep first | keep first | implemented | | slice | keep | drop | | | where | keep | keep | |

cc @shoyer (following up on https://github.com/pydata/xarray/issues/1586#issuecomment-334954046)

The current behavior of the xarray's Groupby.groups property provides a standard (unordered) dictionary. This is fine for most cases but leads to odd orderings in use cases like this one where I am using xarray's FacetGrid plotting:

``` Python plot_kwargs = dict(col='season', vmin=15, vmax=35, levels=12, extend='both')

da_obs = ds_obs.SALT.isel(depth=0).groupby('time.season').mean('time') da_obs.plot(**plot_kwargs) ```

Note that MAM and JJA are out of order.

I think this could be easily fixed by using an OrderedDict in xarray.core.Groupby.groups.

Following @shoyer's comment, it will be pretty straightforward to support creating dask masked arrays within the to_masked_array method. My thought would be that data arrays use dask, would be converted to dask masked arrays, rather than to numpy arrays as they are currently.

Two kinks:

1) The dask masked array feature requires dask 0.15.3 or newer. 2) I'm not sure how to test if an object is a dask.array.ma.MaskedArray (Dask doesn't have a MaskedArray class). @mrocklin - thoughts?