What's the advantage of this over the transpose call?

I would say very little. I'm a lazy programmer.

I appreciate the arguments here of not blending methods which could make them more prone to bugs. I doubt there's any noticeable overhead of computation cost with the extra transpose call.

Welcome to close.

We could follow the axis kwarg to DataArray.expand_dims as precedent. It specifies the axis number for the new dimension.

Thanks. Made a PR before seeing your comment. Welcome to chime in on the PR

Dup of https://github.com/pydata/xarray/pull/6067 but this have a clearer title. Same solution so I don’t mind which one goes in

Thanks for trying out! I have installed xarray with conda in a newly conda environment (both python 3.6 and 3.8), but I still receive the error:

@raybellwaves which python version are you using?

Output of xr.open_dataset("http://opendap.tudelft.nl/thredds/dodsC/IDRA/2019/10/01/IDRA_2019-10-01_11-00_raw_data.nc")

I'm on 3.9 sorry I can do a full list of my env as I pull from an internal source but the core ones around netcdf I see

netcdf-fortran 4.5.3
netcdf4 1.5.7 libnetcdf 4.8.1

Works in the pangeo docker (https://github.com/pangeo-data/pangeo-docker-images/blob/master/pangeo-notebook/environment.yml) if that helps

docker run -it pangeo/pangeo-notebook /bin/bash python import xarray as xr xr.open_dataset("http://opendap.tudelft.nl/thredds/dodsC/IDRA/2019/10/01/IDRA_2019-10-01_11-00_raw_data.nc")

Seemed ok for me. You could try installing with conda: http://xarray.pydata.org/en/stable/getting-started-guide/installing.html#instructions

I see this is probably intentional

return zstore https://github.com/pydata/xarray/blob/main/xarray/backends/api.py#L1436

See https://github.com/pydata/xarray/blob/main/xarray/core/dataset.py#L2022

for how to link the doc string to other parts of the docs

See https://github.com/pydata/xarray/discussions/5689 for reference to this PR

test currently failing with zarr==2.4 in min-all-deps (https://github.com/pydata/xarray/blob/main/ci/requirements/py37-min-all-deps.yml#L49) storage_options was added to zarr in 2.5

still need to think about a test

Friendly ping on a review.

test currently failing https://github.com/pydata/xarray/blob/main/xarray/tests/test_backends.py#L5117

========================================================= short test summary info ========================================================== FAILED xarray/tests/test_backends.py::test_open_fsspec - AttributeError: 'bytes' object has no attribute 'items'

Outside of this PR as it on this previous PR.

Closing as it works for one bucket but not another. Could be a bucket policy thing.

Just came across some of the xr.backends.PydapDataStore stuff here: http://xarray.pydata.org/en/stable/user-guide/io.html#opendap

Haven't tested on the above example

FYI larger issue is open at https://github.com/pydata/xarray/issues/5082

Works. Thanks @joshmoore !

I tend to do ds["var"].encoding = {} before saving. See also https://github.com/pydata/xarray/discussions/5407

FWIW. This showed up in xskillscore as we were doing np.clip(xarray object, min, min). We updated the code to do xarray object.clip(min, max) which we probably should have been doing in the first place (https://github.com/xarray-contrib/xskillscore/pull/309#issue-634256273)

This exists in the current docs

Not sure if https://github.com/conda-forge/python-eccodes-feedstock/pull/71 helps Came across this via https://github.com/ecmwf/cfgrib/issues/226

Thanks. Works great. Closing.

Thanks @Illviljan. I missed that issue. Is the syntax above off slightly? Is it in the docs as well?

Thanks for the FYI. Guessed it would go one way or the other.

Works ok for me on my mac ``` conda create -n test_env python=3.8 --y conda activate test_env conda install -c conda-forge xarray cfgrib ipython wget http://download.ecmwf.int/test-data/cfgrib/era5-levels-members.grib

ipython import xarray as xr ds = xr.open_dataset('era5-levels-members.grib', engine='cfgrib') ```

@skgbanga can this be closed on latest(s) xarray, zarr, fsspec, gcsfs?

Any interest in adding this to docs similar to https://docs.dask.org/en/latest/support.html I'm also fine if it's purpose is just for core devs

Edit: Copied and pasted from a duplicate issue I opened. Closing that and moving convo here.

@jhamman's SO answer circa 2018 helped me this week https://stackoverflow.com/a/51714004/6046019

I wonder if it's worth (not sure where) providing an example of how to use preprocesses with open_mfdataset?

Add an Examples entry to the doc string? (http://xarray.pydata.org/en/latest/generated/xarray.open_mfdataset.html / https://github.com/pydata/xarray/blob/5296ed18272a856d478fbbb3d3253205508d1c2d/xarray/backends/api.py#L895)

While not a small example (as the remote files are large) this is how I used it:

``` import xarray as xr import s3fs

def preprocess(ds): return ds.expand_dims('time')

fs = s3fs.S3FileSystem(anon=True) f1 = fs.open('s3://fmi-opendata-rcrhirlam-surface-grib/2021/02/03/00/numerical-hirlam74-forecast-MaximumWind-20210203T000000Z.grb2') f2 = fs.open('s3://fmi-opendata-rcrhirlam-surface-grib/2021/02/03/06/numerical-hirlam74-forecast-MaximumWind-20210203T060000Z.grb2')

ds = xr.open_mfdataset([f1, f2], engine="cfgrib", preprocess=preprocess, parallel=True) ```

with one file looking like: xr.open_dataset("LOCAL_numerical-hirlam74-forecast-MaximumWind-20210203T000000Z.grb2", engine="cfgrib") <xarray.Dataset> Dimensions: (latitude: 947, longitude: 5294, step: 55) Coordinates: time datetime64[ns] ... * step (step) timedelta64[ns] 01:00:00 ... 2 days 07:00:00 heightAboveGround int64 ... * latitude (latitude) float64 25.65 25.72 25.78 ... 89.86 89.93 90.0 * longitude (longitude) float64 -180.0 -179.9 -179.9 ... 179.9 180.0 valid_time (step) datetime64[ns] ... Data variables: fg10 (step, latitude, longitude) float32 ... Attributes: GRIB_edition: 2 GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 Conventions: CF-1.7 institution: European Centre for Medium-Range Weather Forecasts history: 2021-02-12T18:06:52 GRIB to CDM+CF via cfgrib-0....

A smaller example could be (WIP; note I was hoping ds would concat along t but it doesn't do what I expect) ``` import numpy as np import xarray as xr

f1 = xr.DataArray(np.arange(2), coords=[np.arange(2)], dims=["a"], name="f1") f1 = f1.assign_coords(t=0) f1.to_dataset().to_zarr("f1.zarr") # What's the best way to store small files to open again with mf_dataset? csv via xarray objects? can you use open_mfdataset on pkl objects?

f2 = xr.DataArray(np.arange(2), coords=[np.arange(2)], dims=["a"], name="f2") f2 = f2.assign_coords(t=1) f2.to_dataset().to_zarr("f2.zarr")

Concat along t

def preprocess(ds): return ds.expand_dims('t') ds = xr.open_mfdataset(["f1.zarr", "f2.zarr"], engine="zarr", concat_dim="t", preprocess=preprocess)

ds <xarray.Dataset> Dimensions: (a: 2, t: 1) Coordinates: * t (t) int64 0 * a (a) int64 0 1 Data variables: f1 (t, a) int64 dask.array<chunksize=(1, 2), meta=np.ndarray> f2 (t, a) int64 dask.array<chunksize=(1, 2), meta=np.ndarray> ```

I think my Q on SO is related to this PR https://stackoverflow.com/questions/66145459/open-mfdataset-on-remote-zarr-store-giving-zarr-errors-groupnotfounderror

Was looking at reading a remote zarr store using open_mfdataset

@martindurant suggested putting the single "file" (mapping) in a list which works ds = xr.open_mfdataset([file], engine="zarr")

but I also wanted to test the other suggestion

ds = xr.open_mfdataset(uri, engine="zarr", backend_kwargs=dict(storage_options={'anon': True}))

On current xarray master I get Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/ray.bell/Documents/PYTHON_dev/xarray/xarray/backends/api.py", line 884, in open_mfdataset raise OSError("no files to open") OSError: no files to open

On this branch it works ``` git clone https://github.com/martindurant/xarray.git git checkout fsspec_mk2 conda create -c conda-forge -n xarray-tests python=3.8 conda env update -f ci/requirements/environment.yml conda activate xarray-tests pip install -e . pip install s3fs

import xarray as xr uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr" ds = xr.open_mfdataset(uri, engine="zarr", backend_kwargs=dict(storage_options={'anon': True})) ```

Not sure if I can do anything for the doctest failing https://dev.azure.com/xarray/xarray/_build/results?buildId=4491&view=logs&j=e1da92ae-4a54-5c20-3ee1-ef7b06ffcd80&t=5640f7da-bb07-517b-ec85-0b3cfe0ccaa3&l=62 - * y (y) int32 10 20 30 + * y (y) int64 10 20 30

I can add to the Whats-new. Not sure how to "update of the doctest output"

More than welcome to push to mine.

I would also run pre-commit run --files doc/combining.rst xarray/core/concat.py once, but that's optional.

I'm doing this PR in my windows machine unfortunately, and pre-commit is a pain on windows. Hopefully it's fine as is.

Thanks for the feedback.

It was a copy-paste from http://xarray.pydata.org/en/stable/combining.html#concatenate

I'll implement your suggestions in https://github.com/pydata/xarray/blob/master/doc/combining.rst as well.

i.e. use arange for the creation and isel for the index.

Why not pandas.date_range

Good point. Given the "normal" spacing of the data that makes sense.

The problematic variable in this dataset is "tau"

Ah thanks for the digging into this for me.

Which functionality are you looking for in xarray that pandas Timestamp objects provide, but cftime objects do not?

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.tz_localize.html

For a bit more reference. I'm working with HyCOM data and want to keep track of 'local time'.

ds = xr.open_dataset('https://tds.hycom.org/thredds/dodsC/GLBy0.08/latest', decode_times=False) reference_date = ds.time.attrs['units'][12:25] ds['time'] = xr.cftime_range(start=reference_date, periods=len(ds.time), freq='3H')

Tested the docs build:

Looks as though it does

``` ray@ray-MS-7B43:~$ conda install -c conda-forge cfgrib Collecting package metadata (current_repodata.json): done Solving environment: done

Package Plan

environment location: /home/ray/local/bin/anaconda3/envs/rcclwx-dev

added / updated specs: - cfgrib

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
cffi-1.14.0                |   py38he30daa8_1         225 KB
cfgrib-0.9.8.2             |             py_0          42 KB  conda-forge
eccodes-2.17.0             |       hf05d9b7_2         3.9 MB  conda-forge
------------------------------------------------------------
                                       Total:         4.2 MB

The following NEW packages will be INSTALLED:

attrs conda-forge/noarch::attrs-19.3.0-py_0 cffi pkgs/main/linux-64::cffi-1.14.0-py38he30daa8_1 cfgrib conda-forge/noarch::cfgrib-0.9.8.2-py_0 eccodes conda-forge/linux-64::eccodes-2.17.0-hf05d9b7_2 jasper conda-forge/linux-64::jasper-1.900.1-h07fcdf6_1006 libaec conda-forge/linux-64::libaec-1.0.4-he1b5a44_1 pycparser conda-forge/noarch::pycparser-2.20-py_0 ```

May be able to drop the suggestion of installing eccodes from the docs.

Sure. Can update the docs. but won't be able to for a week or so.

Curious is there is much/any difference? is worth deprecating and pointing maintenance/development to the pandas version instead?

LGTM

@fujiisoup I would like to add another example on how to update a DataArray with the interpolated data. I looked in http://xarray.pydata.org/en/stable/combining.html but couldn't work out how best to do it.

I tried using merge but it requires the DataArray's to be named.

I used concat in this example but it is not correct as I would like the coordinates to be in order (in-place may be the right word).

``` import xarray as xr import pandas as pd import numpy as np

da_dt64 = xr.DataArray([1, 3], [('time', pd.date_range('1/1/2000', '1/3/2000', periods=2))]) a = da_dt64.interp(time=np.datetime64('2000-01-02')) xr.concat([da_dt64, a], dim='time')

<xarray.DataArray (time: 3)>

array([ 1., 3., 2.])

Coordinates:

* time (time) datetime64[ns] 2000-01-01 2000-01-03 2000-01-02

```

I may pick up this PR but will need some hand holding as i'm not familiar with dask.

I want to make sure I can do periodic rolling with dask in a simple script first but it needs work: https://gist.github.com/raybellwaves/621faaf195c0f4ed010b7b0dfee8605a

I was going to suggest this feature so glad others are interested.

In my use case I would like to smooth a daily climatology. My colleague uses matlab and uses https://www.mathworks.com/matlabcentral/fileexchange/52688-nan-tolerant-fast-smooth

Using the slice solution as @mathause showed above, it would look something like (using code from http://xarray.pydata.org/en/stable/examples/weather-data.html#toy-weather-data)

``` import numpy as np import pandas as pd import xarray as xr

times = pd.date_range('2000-01-01', '2010-12-31', name='time') annual_cycle = np.sin(2 * np.pi * (times.dayofyear.values / 366 - 0.28)) noise = 15 * np.random.rand(annual_cycle.size) data = 10 + (15 * annual_cycle) + noise da = xr.DataArray(data, coords=[times], dims='time')

da.plot()

Check variability at one day

da.groupby('time.dayofyear').std('time')[0]

da_clim = da.groupby('time.dayofyear').mean('time') _da_clim = xr.concat([da_clim[-15:], da_clim, da_clim[:15]], 'dayofyear') da_clim_smooth = _da_clim.rolling(dayofyear=31, center=True).mean().dropna('dayofyear')

da_clim_smooth.plot()

```

Thanks for the quick fix @fujiisoup !

I realized there with an issue before that in without center=True it doesn't raise an issue but it returns rubbish:

``` import xarray as xr import numpy as np

a = xr.DataArray(np.arange(1,4), coords=[np.arange(1,4)], dims='x') print(a.rolling(x=3, center=True).mean()) Out[2]: <xarray.DataArray (x: 3)> array([ nan, 2., nan]) Coordinates: * x (x) int64 1 2 3 print(a.chunk().rolling(x=3).mean().values) Out[3]: array([ -6.14891469e+18, -9.22337204e+18, -1.22978294e+19]) ```

The culprit lies in a dask function https://github.com/pydata/xarray/blob/6d8ac11ca0a785a6fe176eeca9b735c321a35527/xarray/core/dask_array_ops.py#L24

Not sure if this is an issue with the function or the way the data is going into the function.

For the center=True issue: https://github.com/pydata/xarray/blob/6d8ac11ca0a785a6fe176eeca9b735c321a35527/xarray/core/rolling.py#L308 is slicing the data.

Probably isn't a good first issue but I wouldn't like to spend some time on this. Welcome to suggest places to look and things to try.

Snuck in an .assign_coords() example. I often use climate data which has longitude values 0-359. I then may subset the data crossing the meridian. Making the longitude values from -180 - 180 allows an easy look at the data using .plot(). My example is based on the example here: https://gis.stackexchange.com/questions/205871/xarray-slicing-across-the-antimeridian

Closes https://github.com/pydata/xarray/issues/2102

For rolling I just realized there is good section in the docs on the use of .construct() and .reduce(). Perhaps in the docstring of rolling I can point to http://xarray.pydata.org/en/stable/computation.html#rolling-window-operations although not sure where this would go in the docstring? Examples? See Also?

I opened up an issue on resample to use the new syntax with an n-d array. Once That's squared up i'll drop the 2-d example with the old syntax https://github.com/pydata/xarray/issues/2102

@fmaussion Agreed. Changed the URLs to link to the latest version of the online docs.

Apologies. Closing. I thought it was a rouge line as it was not declared to a variable and I couldn't see that it was being using anywhere. But I was not familiar with np.random.seed but I am now.

I started to have a look at this.

I needed to install rasterio to build the docs but unfortunately the conda installation isn't working (see https://github.com/conda-forge/rasterio-feedstock/issues/53) nor is the pip installation (see https://github.com/mapbox/rasterio/issues/1244).

Anyone who uses a Mac and mostly installs via conda have any advice installing rasterio?

Would you mind printing md first to show what it looks like?

Apologies. Should have had a better looked at how open_dataset was laid out. Thanks for baring with me

@jhamman it was just a quick comparison with the docs online

see http://xarray.pydata.org/en/stable/generated/xarray.open_dataset.html vs http://xarray.pydata.org/en/stable/generated/xarray.open_dataarray.html

The stickler in me wants to ensure the docs are the same for both functions (i.e. list the inputs are the same) between open_dataset and open_dataarray. I thought it was just a 2 second copy and paste fix but my PR, but it's failing so i'll take another look.

No problem. Feel free to add 'novice' labels to any issues that require grunt work (such as documentation) and i'll be happy to have a stab at them to get more familiar with xarray

Thanks for your help with this. Agree with all of your suggestions.

For the HPC I used pip install numpy scipy netCDF4 xarray Yes this is linux.

For my laptop I used conda install xarray This is a MacBook Pro

@shoyer thanks for looking into this.

Firstly, I updated the hs netCDF file as it was a different size then expected: (1, 82, 131) instead of (1, 81, 131). I've corrected this now but I am still getting an error message.

I've added the version numbers and outputs to https://github.com/raybellwaves/xarray_issue/blob/master/xr_where_issue.py

I have two setups. I run my model experiments and do most of my analysis on a HPC. However, I do most of my plotting on my laptop for ease of viewing. To help me debug this i've run the code on both setups (which each have different python version etc.) but a priority is the more up to date HPC python version. I've including the output from both though.