Is your feature request related to a problem?

Suppose I'm trying to read a large number of netCDF files with open_mfdataset.

Now suppose that one of those files is for some reason incorrect -- for instance there was a problem during the creation of that particular file, and its file size is zero, or it is not valid netCDF. The file exists, but it is invalid.

Currently open_mfdataset will raise an exception with the message ValueError: did not find a match in any of xarray's currently installed IO backends

As far as I can tell, there is currently no way to identify which one(s) of the files being read is the source of the problem. If there are several hundreds of those, finding the problematic files is a task by itself, even though xarray probably knows them.

Describe the solution you'd like

It would be most useful to this particular user if the error message could somehow identify the file(s) responsible for the exception.

Apart from better reporting, I would find it very useful if I could pass to open_mfdataset some kind of argument that would make it ignore invalid files altogether (ignore_invalid=False comes to mind).

Describe alternatives you've considered

No response

Additional context

No response

I am finding that groupby_bins is 10x slower than numpy in what I consider to be a simple implementation.

In the screenshot below, you can see me opening a netCDF file containing two variables with the same single dimension. One variable is the latitude. I want to aggregate (sum) the other variable in bins of latitude. The xarray approach using groupby_bins takes ~314ms per loop, the numpy approach less than 30ms per loop.

I need to do this kind of computation on many more variables, on data spanning several years, and following the xarray approach leads to many more hours of processing :-/

Am I doing something wrong here?

In another issue I create a simple dataset like so:

python lat = np.random.rand(50000) * 180 - 90 lon = np.random.rand(50000) * 360 - 180 d = xr.Dataset({'latitude':lat, 'longitude':lon})

I expected d to contain two variables (latitude and longitude) with no coordinates. Instead d appears to contain two coordinates and no variables:

In [5]: d Out[5]: <xarray.Dataset> Dimensions: (latitude: 50000, longitude: 50000) Coordinates: * latitude (latitude) float64 -76.0 -84.36 26.69 66.44 -37.85 50.13 ... * longitude (longitude) float64 -148.7 -74.82 18.37 117.7 80.63 12.25 ... Data variables: *empty*

Is this desired behavior?

I have a data 3D DataArray with Time, Latitude and Longitude coordinates.

I want to plot maps of this dataset, faceted by Time. The following code

import cartopy.crs as ccrs proj = ccrs.PlateCarree() data.plot(transform=proj, col='Time', col_wrap=3, robust=True)

fails with

ValueError: Axes should be an instance of GeoAxes, got <class 'matplotlib.axes._subplots.AxesSubplot'>

this is because to plot with a transform, the axes must be a GeoAxes, which is done with something like plt.subplot(111, projection=proj). The implicit subplotting done when faceting does not do that. To make the faceting works, I had to do

import cartopy.crs as ccrs proj = ccrs.PlateCarree() data.plot(transform=proj, col='Time', col_wrap=3, robust=True, subplot_kws={'projection':proj})

I propose that, when plot faceting is requested with a transform kw, the content of that keyword should be passed to the subplot function as a projection argument automatically by default. If a projection is provided explicitely like in the call above, use that one.

I have a bunch of netCDF files that I want to read through open_mfdataset. Each file was created with xarray through to_netcdf() and contains several variables with a single time dimension. Most files look like this:

``` netcdf CAL_LID_L2_05kmCLay-Standard-V4-10.2008-07-01T01-54-46ZD.hdf_extract { dimensions: time = 112 ; variables: float lat(time) ; lat:_FillValue = NaNf ; float lon(time) ; lon:_FillValue = NaNf ; float elev(time) ; elev:_FillValue = NaNf ; double daynight(time) ; daynight:_FillValue = NaN ; double surf(time) ; surf:_FillValue = NaN ;

// global attributes: :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17" ; } ```

if one of the files is empty, i.e. the length of the 'time' dimension is zero:

``` netcdf CAL_LID_L2_05kmCLay-Standard-V4-10.2008-01-01T00-37-48ZD.hdf_extract { dimensions: time = UNLIMITED ; // (0 currently) variables: float lat(time) ; lat:_FillValue = NaNf ; float lon(time) ; lon:_FillValue = NaNf ; float elev(time) ; elev:_FillValue = NaNf ; double daynight(time) ; daynight:_FillValue = NaN ; double surf(time) ; surf:_FillValue = NaN ;

// global attributes: :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17" ; } ```

then open_mfdataset crashes with

python File "./test_map_elev_month_gl2.py", line 22, in main data = xr.open_mfdataset(files, concat_dim='time', autoclose=True) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 505, in open_mfdataset **kwargs) for p in paths] File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 505, in <listcomp> **kwargs) for p in paths] File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 301, in open_dataset return maybe_decode_store(store, lock) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/backends/api.py", line 243, in maybe_decode_store lock=lock) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/core/dataset.py", line 1094, in chunk for k, v in self.variables.items()]) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/core/dataset.py", line 1094, in <listcomp> for k, v in self.variables.items()]) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/core/dataset.py", line 1089, in maybe_chunk return var.chunk(chunks, name=name2, lock=lock) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/xarray/core/variable.py", line 540, in chunk data = da.from_array(data, chunks, name=name, lock=lock) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/dask/array/core.py", line 1798, in from_array chunks = normalize_chunks(chunks, x.shape) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/dask/array/core.py", line 1758, in normalize_chunks for s, c in zip(shape, chunks)), ()) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/dask/array/core.py", line 1758, in <genexpr> for s, c in zip(shape, chunks)), ()) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/dask/array/core.py", line 881, in blockdims_from_blockshape for d, bd in zip(shape, chunks)) File "/home/noel/.conda/envs/python3/lib/python3.6/site-packages/dask/array/core.py", line 881, in <genexpr> for d, bd in zip(shape, chunks)) ZeroDivisionError: integer division or modulo by zero

this catches transform kw passed to plot() when faceting, and pass the associated projection to the subplots. This does what was suggested in issue #1201.

(sorry for the irrelevant change to reshaping.rst, it seems I've stuck myself in a git hole and can't get out)

This change explains - how to pass projection arguments to faceted subplots - how to access the Axes of the created subplots (not relevant only to cartopy), relevant to issue #1202

First I create some fake latitude and longitude points. I stash them in a dataset, and compute a 2d histogram on those.

```python

!/usr/bin/env python

import xarray as xr import numpy as np

lat = np.random.rand(50000) * 180 - 90 lon = np.random.rand(50000) * 360 - 180 d = xr.Dataset({'latitude':lat, 'longitude':lon})

latbins = np.r_[-90:90:2.] lonbins = np.r_[-180:180:2.] h, xx, yy = np.histogram2d(d['longitude'], d['latitude'], bins=(lonbins, latbins)) ```

When I run this I get some underwhelming performance:

```

time ./test_with_xarray.py

real 0m28.152s user 0m27.201s sys 0m0.630s ```

If I change the last line to

python h, xx, yy = np.histogram2d(d['longitude'].values, d['latitude'].values, bins=(lonbins, latbins))

(i.e. I pass the numpy arrays directly to the histogram2d function), things are very different:

```

time ./test_with_xarray.py

real 0m0.996s user 0m0.569s sys 0m0.253s ```

It's ~28 times slower to call histogram2d on the DataArrays, compared to calling it on the underlying numpy arrays. I ran into this issue while histogramming quite large lon/lat vectors from multiple netCDF files. I got tired waiting for the computation to end, added the .values to the call and went through very quickly.

It seems problematic that using xarray can slow down your code by 28 times with no real way for you to know about it...

when using a DataArray plot function, it is already possible to specify the figure size and aspect ratio.

I think it would make sense to also be able to specify the dpi, e.g. x.plot(dpi=109). It would save one call to plt.figure().

As the title says :-)

This would be very useful to downsample long time series read from multiple consecutive netcdf files.

Note that I was able to apply the rolling window by converting my variable to a pandas series with to_series(). I then could use panda's own rolling window methods. I guess that when converting to a pandas series the dask array is read in memory?

• [x] closes #1290

Say I have a DataArray zwith dimensions ('x', 'y'), in that order. If I call z.plot(), it will create a pcolormesh with x the vertical dimension and y the horizontal one. If I want to invert the axes, I need to call z.plot(x='x', y='y'). If I supply only one of the dimensions, i.e. z.plot(x='x'), I get the error message

python ValueError: cannot supply only one of x and y

I think that when calling the plot function on a 2d DataArray and passing only one coordinate, as in z.plot(x='x'), xarray could guess we want the unpassed coordinate to be the other one. I don't see why this would be unsafe.

When plotting maps with cartopy, it is common to request plotting additional information over the map, e.g. coastlines using ax.coastlines().

When faceting maps (as in issue #1201), AFAICS there is no way to add coastlines to each of the faceted subplots. I do not know if or how this can be done, but in my view not being able to do it severely limits the interest of faceting when dealing with maps.

sorry if I messed up, I'm not a github master