• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

I wanted to know what zarr.convenience.consolidate_metadata does which motivated me to add a link to it in the docstring to make it easier for others (and future me) to discover it.

I sorted intersphinx_mapping to make it easier to docs from other projects. I also added :func:zarr.convenience.consolidate_metadata in the doc string.

Testing if it actually works is WIP

Is your feature request related to a problem?

I was trying to store a zarr fstore and I got:

/opt/userenvs/ray.bell/autofront/lib/python3.9/site-packages/xarray/core/dataset.py:2060: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs return to_zarr( # type: ignore

~~What I was seeing was a time coord which had encoding 'dtype': dtype('int64'). Storing the time this way lost the HH:MM on my datetime array.~~ I think I am seeing 'units': 'days since 2022-MM-DD 00:00:00' whereas it should be hours for my data.

Describe the solution you'd like

The message captures the variable/coord name instead of None

It could offer a solution such as "to store floating point data you could possibly use encoding={variable/coord: {"dtype": float, "_Fillvalue": np.nan}"

Describe alternatives you've considered

No response

Additional context

I'll try to come back and write a MCVE

WIP: you can get the warning message by doing: ds = xr.tutorial.open_dataset("air_temperature") ds["time"].encoding # {'units': 'hours since 1800-01-01', 'calendar': 'standard', 'dtype': dtype('float32')} ds.isel(time=0).to_zarr("tmp0.zarr", mode="w") /Users/ray/miniconda3/envs/main/lib/python3.9/site-packages/xarray/core/dataset.py:2060: SerializationWarning: saving variable None with floating point data as an integer dtype without any _FillValue to use for NaNs return to_zarr( # type: ignore

My guess if one of lat, lon, time, air is raising that message

Can't find the associated code in the repo that raises this. 2060 is to_zarr AFICT (https://github.com/pydata/xarray/blob/main/xarray/core/dataset.py#L2066)

Is your feature request related to a problem? Please describe. In the docs (http://xarray.pydata.org/en/stable/generated/xarray.Dataset.dropna.html) it says Dropping along multiple dimensions simultaneously is not yet supported. But I couldn't find a GitHub Issue open it. Apologies if there is one

Describe the solution you'd like da.dropna(['lat', 'lon], how="all") produces the same as da.dropna('lat', how="all").dropna('lon', how="all")

What happened:

When doing to zarr it returns <xarray.backends.zarr.ZarrStore at ...>

What you expected to happen:

Returns None. Same behaviour as pandas.to_parquet()

Minimal Complete Verifiable Example:

```python import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature") ds.to_zarr("ds.zarr")

ds.to_dataframe().to_parquet("ds.parquet") ```

Anything else we need to know?:

Environment:

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Help navigation in the docs of going between the DataArray method and the Dataset method.

• [x] Closes #6754
• [x] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Is your feature request related to a problem?

I do ds.to_array(dim="variable").transpose("latitude", "longitude", "variable"). I would like to avoid the extra transpose call.

Describe the solution you'd like

ds.to_array(dim="variable", new_axis="last") where new_axis is Literal["first", "last"] = "first" or ds.to_array(dim="variable", new_axis=0) where new_axis is Literal[0, -1] = 0

code to change: https://github.com/pydata/xarray/blob/main/xarray/core/dataset.py#L5770

Describe alternatives you've considered

No response

Additional context

I imagine new_axis could be of type int to place the new axis where you would like but the proposal above may be a good first step.

For reference, i'm doing deep learning and want to shape the data as width (latitude), height (longitude), channel (feature)

Some tiny overlap with https://github.com/pydata/xarray/pull/6542.

I feel this minor changes helps it easy to navigate to other parts of the docs.

Some motivation behind the PR here: https://github.com/pydata/xarray/discussions/6697

Note: i've forgotten the syntax for how to create a link to https://xarray.pydata.org/en/stable/generated/xarray.Dataset.chunk.html

Is your feature request related to a problem?

I tend to use xr.cftime_range as a drop in for pandas.date_range. My only reason here is I may able to shave a line and not import pandas.

In doing xr.cftime_range("2020-01-01", periods=1, freq="W") I get

``` AttributeError Traceback (most recent call last) /opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftime_offsets.py in to_offset(freq) 729 try: --> 730 freq_data = re.match(_PATTERN, freq).groupdict() 731 except AttributeError:

AttributeError: 'NoneType' object has no attribute 'groupdict'

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) /tmp/ipykernel_549/3579801314.py in <module> ----> 1 xr.cftime_range("2020-01-01", periods=1, freq="W")

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftime_offsets.py in cftime_range(start, end, periods, freq, normalize, name, closed, calendar) 1041 dates = _generate_linear_range(start, end, periods) 1042 else: -> 1043 offset = to_offset(freq) 1044 dates = np.array(list(_generate_range(start, end, periods, offset))) 1045

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftime_offsets.py in to_offset(freq) 730 freq_data = re.match(_PATTERN, freq).groupdict() 731 except AttributeError: --> 732 raise ValueError("Invalid frequency string provided") 733 734 freq = freq_data["freq"]

ValueError: Invalid frequency string provided ```

In pandas I get

```

pd.date_range("2020-01-01", periods=1, freq="W") DatetimeIndex(['2020-01-05'], dtype='datetime64[ns]', freq='W-SUN') ```

Describe the solution you'd like

No response

Describe alternatives you've considered

Closing this issue as outside of scope

Additional context

No response

Kind of related to https://github.com/pydata/xarray/issues/4697

I see https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html#rioxarray

shows

ds = xarray.open_dataset("file.tif", engine="rasterio")

This could be added to

https://xarray.pydata.org/en/latest/user-guide/io.html#rasterio

Is your feature request related to a problem?

For better or worse, I use xr.cftime_range interchangably with pd.date_range, mostly it is saves my importing pandas.

Pandas allows a string named "now" to be parsed in pd.date_range pd.date_range("2000-01-01", "now", freq="H") It would be nice if xr.cftime_range could do the same (see additional context for Traceback).

Describe the solution you'd like

Something like assert pd.date_range("2000-01-01", "now", freq="H") == xr.cftime_range("2000-01-01", "now", freq="H")

Describe alternatives you've considered

Close as not within the scope of xr.cftime_range

Additional context

xr.cftime_range("2000-01-01", "now", freq="H")

```python-traceback ValueError Traceback (most recent call last) /tmp/ipykernel_15451/3415490845.py in <module> ----> 1 xr.cftime_range("2000-01-01", "now", freq="H")

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftime_offsets.py in cftime_range(start, end, periods, freq, normalize, name, closed, calendar) 982 start = _maybe_normalize_date(start, normalize) 983 if end is not None: --> 984 end = to_cftime_datetime(end, calendar) 985 end = _maybe_normalize_date(end, normalize) 986

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftime_offsets.py in to_cftime_datetime(date_str_or_date, calendar) 697 "a calendar type must be provided" 698 ) --> 699 date, _ = _parse_iso8601_with_reso(get_date_type(calendar), date_str_or_date) 700 return date 701 elif isinstance(date_str_or_date, cftime.datetime):

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftimeindex.py in _parse_iso8601_with_reso(date_type, timestr) 125 126 default = date_type(1, 1, 1) --> 127 result = parse_iso8601_like(timestr) 128 replace = {} 129

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/coding/cftimeindex.py in parse_iso8601_like(datetime_string) 115 if match: 116 return match.groupdict() --> 117 raise ValueError( 118 f"no ISO-8601 or cftime-string-like match for string: {datetime_string}" 119 )

ValueError: no ISO-8601 or cftime-string-like match for string: now ```

• [ ] Closes #xxxx
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

In the section above (http://xarray.pydata.org/en/stable/contributing.html#how-to-build-the-xarray-documentation) i.e. build the env, it assumes you are in the root directory. This holds that assumptions and makes it easier for a user to copy-paste to build the docs

Related to https://github.com/pydata/xarray/issues/4001#issuecomment-779556647

When I worked on https://github.com/pydata/xarray/pull/5615 I did so with a couple of team mates. I added them in the commit message: https://github.com/pydata/xarray/pull/5615/commits/67ccce2f9b31eebf38fffb5dc86b90810299111c

Want to make sure they get a shout out here :)

What happened:

Took engine="zarr" out of my open_mfdataset call and got

``` ValueError Traceback (most recent call last) /tmp/ipykernel_24527/4212238570.py in <module> 3 TqdmCallback(desc="dask tasks").register() 4 ----> 5 ds = xr.open_mfdataset( 6 "s3://era5-pds/zarr///data/air_temperature_at_2_metres.zarr", 7 parallel=True,

/opt/userenvs/ray.bell/main/lib/python3.9/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, data_vars, coords, combine, parallel, join, attrs_file, combine_attrs, **kwargs) 860 paths = [fs.get_mapper(path) for path in paths] 861 elif is_remote_uri(paths): --> 862 raise ValueError( 863 "cannot do wild-card matching for paths that are remote URLs: " 864 "{!r}. Instead, supply paths as an explicit list of strings.".format(

ValueError: cannot do wild-card matching for paths that are remote URLs: 's3://era5-pds/zarr///data/air_temperature_at_2_metres.zarr'. Instead, supply paths as an explicit list of strings. ```

What you expected to happen:

Give a suggestion that this can be fixed if engine="zarr"

Minimal Complete Verifiable Example:

```python import xarray as xr

ds = xr.open_mfdataset( "s3://era5-pds/zarr/2020/1*/data/eastward_wind_at_10_metres.zarr", backend_kwargs=dict(storage_options={"anon": True}), )

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/ray/miniconda3/envs/main/lib/python3.9/site-packages/xarray/backends/api.py", line 862, in open_mfdataset raise ValueError( ValueError: cannot do wild-card matching for paths that are remote URLs: 's3://era5-pds/zarr/2020/1*/data/eastward_wind_at_10_metres.zarr'. Instead, supply paths as an explicit list of strings. ```

```python ds = xr.open_mfdataset( "s3://era5-pds/zarr/2020/1*/data/eastward_wind_at_10_metres.zarr", backend_kwargs=dict(storage_options={"anon": True}), engine="zarr", )

works

```

Anything else we need to know?:

message here: https://github.com/pydata/xarray/blob/main/xarray/backends/api.py#L861

Environment:

• [x] Closes #5838
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [x] Closes my brain dump in https://github.com/pydata/xarray/discussions/5795
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

• [ ] Tests added
• [x] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst

What does this PR do?

Adds a storage_options arg to to_zarr.

What is the storage_options arg?

The storage_options arg is used throughout the pydata ecosystem where you can write a file to cloud storage. Such as:

• pandas: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_parquet.html?highlight=to_parquet
• dask.dataframe: https://docs.dask.org/en/latest/generated/dask.dataframe.to_parquet.html
• dask.array: https://docs.dask.org/en/latest/generated/dask.array.to_zarr.html#dask.array.to_zarr

It allows you to write to a prd account or a dev account for example.

It is can be used currently in xarray when reading data via open_dataset e.g. xr.open_dataset(..., backend_kwargs=dict(storage_options=storage_options)

Why is it needed?

When working in cloud based environments you storage options are often automatically created. I recently found when doing things in parallel dask can hiccup on finding the credentials (see https://dask.slack.com/archives/C02282J3Q6Q/p1626201210115500) passing the storage options as an arg gave it some stability.

I imagine this could help when writing multiple zarr stores to cloud storage in parallel (e.g. delayed).

This PR also brings similar functionality of pandas.DataFrame.to_parquet and dask.array.to_zarr to xarray.

How is this tested?

Not obvious how to test. e.g. no storage_options test in dask.array https://github.com/dask/dask/blob/main/dask/array/tests/test_array_core.py

Pandas has more extensive tests https://github.com/pandas-dev/pandas/blob/master/pandas/tests/io/test_fsspec.py

Background:

The idea for this PR is discussed here: https://github.com/pydata/xarray/discussions/5601 cc. @martindurant

In http://xarray.pydata.org/en/stable/contributing.html#running-the-test-suite

The suggestion is

Using pytest-xdist, one can speed up local testing on multicore machines. To use this feature, you will need to install pytest-xdist via: pip install pytest-xdist Then, run pytest with the optional -n argument: pytest xarray -n 4

pytest-xdist is in the environment (https://github.com/pydata/xarray/blob/main/ci/requirements/environment.yml#L39) no need to reinstall

Makes life a little easier when navigating the docs

Note: for upstream issue see https://github.com/dask/dask/issues/7887

What happened:

ds.to_zarr("s3://BUCKET/file.zarr") gives

What you expected to happen:

Finds AWS credentials and writes zarr object to AWS S3 cloud storage.

Minimal Complete Verifiable Example:

```python import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature").isel(time=0) ds.to_zarr("s3://BUCKET/file.zarr")

ClientError Traceback (most recent call last) ~/opt/miniconda3/envs/main/lib/python3.9/site-packages/s3fs/core.py in _call_s3(self, method, akwarglist, kwargs) 245 try: --> 246 out = await method(*additional_kwargs) 247 return out

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/aiobotocore/client.py in _make_api_call(self, operation_name, api_params) 153 error_class = self.exceptions.from_code(error_code) --> 154 raise error_class(parsed_response, operation_name) 155 else:

ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

The above exception was the direct cause of the following exception:

PermissionError Traceback (most recent call last) ~/opt/miniconda3/envs/main/lib/python3.9/site-packages/zarr/storage.py in setitem(self, key, value) 1110 self.fs.rm(path, recursive=True) -> 1111 self.map[key] = value 1112 self.fs.invalidate_cache(self.fs._parent(path))

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/fsspec/mapping.py in setitem(self, key, value) 151 self.fs.mkdirs(self.fs._parent(key), exist_ok=True) --> 152 self.fs.pipe_file(key, maybe_convert(value)) 153

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/fsspec/asyn.py in wrapper(args, kwargs) 86 self = obj or args[0] ---> 87 return sync(self.loop, func, args, **kwargs) 88

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/fsspec/asyn.py in sync(loop, func, timeout, args, *kwargs) 67 if isinstance(result[0], BaseException): ---> 68 raise result[0] 69 return result[0]

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/fsspec/asyn.py in _runner(event, coro, result, timeout) 23 try: ---> 24 result[0] = await coro 25 except Exception as ex:

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/s3fs/core.py in _pipe_file(self, path, data, chunksize, kwargs) 865 if size < min(5 * 2 ** 30, 2 * chunksize): --> 866 return await self._call_s3( 867 "put_object", Bucket=bucket, Key=key, Body=data, kwargs

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/s3fs/core.py in _call_s3(self, method, akwarglist, *kwargs) 264 err = e --> 265 raise translate_boto_error(err) 266

PermissionError: Access Denied

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last) /var/folders/rf/26llfhwd68x7cftb1z3h000w0000gp/T/ipykernel_10507/3272073269.py in <module> 1 ds = xr.tutorial.open_dataset("air_temperature").isel(time=0) ----> 2 ds.to_zarr("s3://BUCKET/file.zarr")

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/xarray/core/dataset.py in to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks) 1920 encoding = {} 1921 -> 1922 return to_zarr( 1923 self, 1924 store=store,

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/xarray/backends/api.py in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks) 1432 ) 1433 -> 1434 zstore = backends.ZarrStore.open_group( 1435 store=store, 1436 mode=mode,

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/xarray/backends/zarr.py in open_group(cls, store, mode, synchronizer, group, consolidated, consolidate_on_close, chunk_store, storage_options, append_dim, write_region, safe_chunks) 336 zarr_group = zarr.open_consolidated(store, open_kwargs) 337 else: --> 338 zarr_group = zarr.open_group(store, open_kwargs) 339 return cls( 340 zarr_group, consolidate_on_close, append_dim, write_region, safe_chunks

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/zarr/hierarchy.py in open_group(store, mode, cache_attrs, synchronizer, path, chunk_store, storage_options) 1183 raise ContainsGroupError(path) 1184 else: -> 1185 init_group(store, path=path, chunk_store=chunk_store) 1186 1187 # determine read only status

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/zarr/storage.py in init_group(store, overwrite, path, chunk_store) 486 487 # initialise metadata --> 488 _init_group_metadata(store=store, overwrite=overwrite, path=path, 489 chunk_store=chunk_store) 490

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/zarr/storage.py in _init_group_metadata(store, overwrite, path, chunk_store) 513 meta = dict() # type: ignore 514 key = _path_to_prefix(path) + group_meta_key --> 515 store[key] = encode_group_metadata(meta) 516 517

~/opt/miniconda3/envs/main/lib/python3.9/site-packages/zarr/storage.py in setitem(self, key, value) 1112 self.fs.invalidate_cache(self.fs._parent(path)) 1113 except self.exceptions as e: -> 1114 raise KeyError(key) from e 1115 1116 def delitem(self, key):

KeyError: '.zgroup' ```

missing code formatting on this https://xarray.pydata.org/en/latest/generated/xarray.open_dataset.html

It took me a while to realize this so I wanted to make a note in the docs for myself in the future and others uses.

I was playing with pint-xarray and I couldn't work out why I couldn't round trip a file (https://github.com/xarray-contrib/pint-xarray/issues/109). It turns out the DataArray had an enconding attribute which was being used (and overriding my unit conversion). In addition, I was also scratching my head for a while as to where my zarr store was getting chunks from which didn't correspomd to my numpy array (https://github.com/pydata/xarray/discussions/5407).

This PR just makes a note of the importance of encoding attribute of DataArray(s) when writing to a zarr store.

docs page change is here: https://xray--5427.org.readthedocs.build/en/5427/generated/xarray.Dataset.to_zarr.html?highlight=zarr#xarray.Dataset.to_zarr

This is a paraphrase of a SO Q: https://stackoverflow.com/questions/66178846/read-in-authorized-opendap-url-using-xarray

Is your feature request related to a problem? Please describe. I would like to use xarray to read data stored at UCAR's Research Data Archive. To access data you have to create an account with a username and password.

Describe the solution you'd like url = ""https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2" ds = xr.open_dataset(url, backend_kwargs={"auth": ("username", "password")})

Describe alternatives you've considered You can read this file by create dot files in your home directory (https://stackoverflow.com/a/66179413/6046019)

Additional context The idea of "auth" comes from siphon (https://unidata.github.io/siphon/latest/examples/Basic_Usage.html#sphx-glr-examples-basic-usage-py).

Keewis mentioned other options to merge are concat and combine_by_coords (https://github.com/pydata/xarray/discussions/5462#discussioncomment-861492). This PR adds combine_by_coords in the see also docstring of merge

What happened:

I saved a zarr store to a remote location (s3) and read it again and realized the name of the data variables (DataArrays) had reverted to lower case

What you expected to happen:

This does not happen when you save the zarr store locally

Minimal Complete Verifiable Example:

```python ds = xr.tutorial.open_dataset("air_temperature") ds = ds.rename({'air': "AIR"})

Save to local

ds.to_zarr("ds.zarr")

Save to remote

ds.to_zarr("s3://BUCKET/ds.zarr")

Read local

xr.open_dataset("ds.zarr", engine="zarr").data_vars

Data variables:

AIR (time, lat, lon) float32 ...

Read remote

xr.open_dataset("s3://BUCKET/ds.zarr", engine="zarr").data_vars

Data variables:

air (time, lat, lon) float32 ...

```

Anything else we need to know?:

Environment:

The method xarray.Dataset.to_dask_dataframe exists and would like to make a feature request for the opposite xarray.Dataset.from_dask_dataframe. The conversation started over at dask but it was suggested by @shoyer to implement here first https://github.com/dask/dask/issues/6058

Add query examples in the docstrings.

Relevent rtd buld links: - https://xray--5249.org.readthedocs.build/en/5249/generated/xarray.Dataset.query.html?highlight=query#xarray.Dataset.query - https://xray--5249.org.readthedocs.build/en/5249/generated/xarray.DataArray.query.html?highlight=query#xarray.DataArray.query

Fixes rendering in https://github.com/pydata/xarray/pull/5163

The docs current looks likes below. There are a lot of useful sections in Reading and Writing Files (http://xarray.pydata.org/en/latest/user-guide/io.html). Not sure if it's possible to add a drop down arrow (a third level) to see the sections from the table of contents on the left

https://github.com/pydata/xarray/discussions/5004

• [x] Closes #5159
• [ ] Tests added
• [ ] Passes pre-commit run --all-files
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

the to_zarr docs http://xarray.pydata.org/en/latest/generated/xarray.Dataset.to_zarr.html could add a "See Also" and point to http://xarray.pydata.org/en/latest/user-guide/io.html?highlight=zarr#zarr

https://github.com/pydata/xarray/issues/4992#issuecomment-790685921

idea discussed here: https://github.com/pydata/xarray/discussions/4991

Copying here

the data_vars method provides a convenient ways to see the varaibles and the formatting (https://github.com/pydata/xarray/blob/master/xarray/core/dataset.py#L484) is nice. However, there are times I would look to see the full variables names not just a subset (12 is the default).

Ideally there would be a setting to toggle to specify the length of the formatter akin to https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html#overview

xr.set_option('display.max_data_vars', N)

It would be nice to be able to convert cftime variables to pandas datetime to utilize the functionality there.

I understand this is an upstream issue as pandas probably isn't aware of cftime. However, i'm curious if a method could be added to cftime such as .to_dataframe().

I've found pd.to_datetime(np.datetime64(date_cf)) is the best way to do this currently.

``` import xarray as xr import numpy as np import pandas as pd

date_str = '2020-01-01' date_np = np.datetime64(date_str)

date_np numpy.datetime64('2020-01-01') date_pd = pd.to_datetime(date_np) date_pd Timestamp('2020-01-01 00:00:00')

date_cf = xr.cftime_range(start=date_str, periods=1)[0] pd.to_datetime(date_cf)

pd.to_datetime(date_cf) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/ray/local/bin/anaconda3/envs/a/lib/python3.8/site-packages/pandas/core/tools/datetimes.py", line 830, in to_datetime result = convert_listlike(np.array([arg]), format)[0] File "/home/ray/local/bin/anaconda3/envs/a/lib/python3.8/site-packages/pandas/core/tools/datetimes.py", line 459, in _convert_listlike_datetimes result, tz_parsed = objects_to_datetime64ns( File "/home/ray/local/bin/anaconda3/envs/a/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py", line 2044, in objects_to_datetime64ns result, tz_parsed = tslib.array_to_datetime( File "pandas/_libs/tslib.pyx", line 352, in pandas._libs.tslib.array_to_datetime File "pandas/_libs/tslib.pyx", line 579, in pandas._libs.tslib.array_to_datetime File "pandas/_libs/tslib.pyx", line 718, in pandas._libs.tslib.array_to_datetime_object File "pandas/_libs/tslib.pyx", line 552, in pandas._libs.tslib.array_to_datetime TypeError: <class 'cftime._cftime.DatetimeGregorian'> is not convertible to datetime ```

@jhamman's SO answer circa 2018 helped me this week https://stackoverflow.com/a/51714004/6046019

I wonder if it's worth (not sure where) providing an example of how to use preprocesses with open_mfdataset?

Add an Examples entry to the doc string? (http://xarray.pydata.org/en/latest/generated/xarray.open_mfdataset.html / https://github.com/pydata/xarray/blob/5296ed18272a856d478fbbb3d3253205508d1c2d/xarray/backends/api.py#L895)

While not a small example (as the remote files are large) this is how I used it:

``` import xarray as xr import s3fs

def preprocess(ds): return ds.expand_dims('time')

fs = s3fs.S3FileSystem(anon=True) f1 = fs.open('s3://fmi-opendata-rcrhirlam-surface-grib/2021/02/03/00/numerical-hirlam74-forecast-MaximumWind-20210203T000000Z.grb2') f2 = fs.open('s3://fmi-opendata-rcrhirlam-surface-grib/2021/02/03/06/numerical-hirlam74-forecast-MaximumWind-20210203T060000Z.grb2')

ds = xr.open_mfdataset([f1, f2], engine="cfgrib", preprocess=preprocess, parallel=True) ```

with one file looking like: xr.open_dataset("LOCAL_numerical-hirlam74-forecast-MaximumWind-20210203T000000Z.grb2", engine="cfgrib") <xarray.Dataset> Dimensions: (latitude: 947, longitude: 5294, step: 55) Coordinates: time datetime64[ns] ... * step (step) timedelta64[ns] 01:00:00 ... 2 days 07:00:00 heightAboveGround int64 ... * latitude (latitude) float64 25.65 25.72 25.78 ... 89.86 89.93 90.0 * longitude (longitude) float64 -180.0 -179.9 -179.9 ... 179.9 180.0 valid_time (step) datetime64[ns] ... Data variables: fg10 (step, latitude, longitude) float32 ... Attributes: GRIB_edition: 2 GRIB_centre: ecmf GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts GRIB_subCentre: 0 Conventions: CF-1.7 institution: European Centre for Medium-Range Weather Forecasts history: 2021-02-12T18:06:52 GRIB to CDM+CF via cfgrib-0....

A smaller example could be (WIP; note I was hoping ds would concat along t but it doesn't do what I expect) ``` import numpy as np import xarray as xr

f1 = xr.DataArray(np.arange(2), coords=[np.arange(2)], dims=["a"], name="f1") f1 = f1.assign_coords(t=0) f1.to_dataset().to_zarr("f1.zarr") # What's the best way to store small files to open again with mf_dataset? csv via xarray objects? can you use open_mfdataset on pkl objects?

f2 = xr.DataArray(np.arange(2), coords=[np.arange(2)], dims=["a"], name="f2") f2 = f2.assign_coords(t=1) f2.to_dataset().to_zarr("f2.zarr")

Concat along t

def preprocess(ds): return ds.expand_dims('t') ds = xr.open_mfdataset(["f1.zarr", "f2.zarr"], engine="zarr", concat_dim="t", preprocess=preprocess)

ds <xarray.Dataset> Dimensions: (a: 2, t: 1) Coordinates: * t (t) int64 0 * a (a) int64 0 1 Data variables: f1 (t, a) int64 dask.array<chunksize=(1, 2), meta=np.ndarray> f2 (t, a) int64 dask.array<chunksize=(1, 2), meta=np.ndarray> ```

Update hyperlink for xskillscore

• [x] Closes #4620

Could link http://xarray.pydata.org/en/stable/combining.html#concatenate or add the example to the concat doc string: http://xarray.pydata.org/en/stable/generated/xarray.concat.html

This PR follows a TypeError I got

ds = xr.open_dataset('https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/Best') ds.rename({'lat': 'latitude', 'lon': 'longitude'}, inplace=True) TypeError: The `inplace` argument has been removed from xarray. You can achieve an identical effect with python's standard assignment.

I saw the option of the inplace in the docs http://xarray.pydata.org/en/stable/generated/xarray.Dataset.rename.html

Removing this argument.

TODO:

• [ ] rm test_rename_inplace

I just noticed that doing

da = xr.DataArray(np.random.rand(10, 10, 10), dims=('a', 'b', 'c')) da.to_pandas()

gives

```

KeyError Traceback (most recent call last) ~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/xarray/core/dataarray.py in to_pandas(self) 2397 try: -> 2398 constructor = constructors[self.ndim] 2399 except KeyError:

KeyError: 3

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) <ipython-input-22-2c9fae875981> in <module> ----> 1 da.to_pandas()

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/xarray/core/dataarray.py in to_pandas(self) 2398 constructor = constructors[self.ndim] 2399 except KeyError: -> 2400 raise ValueError( 2401 "cannot convert arrays with %s dimensions into " 2402 "pandas objects" % self.ndim

ValueError: cannot convert arrays with 3 dimensions into pandas objects ```

It is worth raising a suggestion (see below)? Or could you even do da.to_dataset(name='da').to_dataframe() under the hood. I feel the output would be what the user expects but I imagine that is not always true

da.to_dataset(name='da').to_dataframe()

When opening the current url https://github.com/timothycrosley/isort it points to a new url https://github.com/PyCQA/isort.

This PR replaces the old url with the new one

http://xarray.pydata.org/en/stable/generated/xarray.DataArray.to_netcdf.html#xarray.DataArray.to_netcdf

I believe there is a way to hyperlink to xarray.Dataset.to_netcdf. I believe there is also a way for sphinx to render the Note section e.g. in the doc string add .. note::

In the docs http://xarray.pydata.org/en/stable/io.html#grib-format-via-cfgrib

Curious if eccodes is needed if reading a grib using using the cfgrib backend?

Does installing cfgrib via conda also install the binary dependencies? https://github.com/ecmwf/cfgrib#installation

cc. @alexamici

• [x] Closes #4151
• [NA] Tests added
• [NA] Passes isort -rc . && black . && mypy . && flake8
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API

• [X] Closes #3895
• [ ] Tests added
• [ ] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

Tested the docs build

Is it worth referencing https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_xarray.html in http://xarray.pydata.org/en/stable/generated/xarray.Dataset.from_dataframe.html?

is the pandas method preferred?

Copied the install instructions from the docs http://xarray.pydata.org/en/stable/installing.html#instructions on my windows machine. conda install -c conda-forge xarray cartopy pynio pseudonetcdf

Got a PackagesNotFoundError: pynio

Just testing the docs on binder (good job). Noticed the ERA5-Grib-example.ipynb (https://github.com/pydata/xarray/blob/master/doc/examples/ERA5-GRIB-example.ipynb) was not working

```

HTTPError Traceback (most recent call last) <ipython-input-2-783584127f97> in <module> ----> 1 ds = xr.tutorial.load_dataset('era5-2mt-2019-03-uk.grib', engine='cfgrib')

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/tutorial.py in load_dataset(args, kwargs) 107 open_dataset 108 """ --> 109 with open_dataset(args, **kwargs) as ds: 110 return ds.load() 111

/srv/conda/envs/notebook/lib/python3.8/site-packages/xarray/tutorial.py in open_dataset(name, cache, cache_dir, github_url, branch, **kws) 75 76 url = "/".join((github_url, "raw", branch, fullname)) ---> 77 urlretrieve(url, localfile) 78 url = "/".join((github_url, "raw", branch, md5name)) 79 urlretrieve(url, md5file)

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in urlretrieve(url, filename, reporthook, data) 245 url_type, path = _splittype(url) 246 --> 247 with contextlib.closing(urlopen(url, data)) as fp: 248 headers = fp.info() 249

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context) 220 else: 221 opener = _opener --> 222 return opener.open(url, data, timeout) 223 224 def install_opener(opener):

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in open(self, fullurl, data, timeout) 529 for processor in self.process_response.get(protocol, []): 530 meth = getattr(processor, meth_name) --> 531 response = meth(req, response) 532 533 return response

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in http_response(self, request, response) 638 # request was successfully received, understood, and accepted. 639 if not (200 <= code < 300): --> 640 response = self.parent.error( 641 'http', request, response, code, msg, hdrs) 642

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in error(self, proto, args) 567 if http_err: 568 args = (dict, 'default', 'http_error_default') + orig_args --> 569 return self._call_chain(args) 570 571 # XXX probably also want an abstract factory that knows when it makes

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in _call_chain(self, chain, kind, meth_name, args) 500 for handler in handlers: 501 func = getattr(handler, meth_name) --> 502 result = func(args) 503 if result is not None: 504 return result

/srv/conda/envs/notebook/lib/python3.8/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs) 647 class HTTPDefaultErrorHandler(BaseHandler): 648 def http_error_default(self, req, fp, code, msg, hdrs): --> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp) 650 651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 404: Not Found ```

I posted this question here: http://stackoverflow.com/questions/43485347/python-xarray-copy-nan-from-one-dataarray-to-another

I have uploaded the files and code to https://github.com/raybellwaves/xarray_issue.git

When I originally discovered the issue I was using python version 3.6.0 (default, Jan 28 2017, 13:49:14) [GCC Intel(R) C++ gcc 4.4 mode]

I tested the code with version 2.7.13 | packaged by conda-forge | (default, Mar 20 2017, 14:26:36) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] and I did not get the error message which I posted in the question. The code worked by the variable ws10_masked is empty

Think my package should go in the 'extend xarray capabilities' section rather than the 'geoscience' section.

Potentially fixes #1115 as I have a correlation function. http://xarray.pydata.org/en/stable/dask.html?highlight=spearman#automatic-parallelization probably fixes #1115 as well

• [NA] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [NA] Tests added (for all bug fixes or enhancements)
• [NA] Tests passed (for all non-documentation changes)
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

I wanted to add an example as googling 'xarray interp' takes me to http://xarray.pydata.org/en/stable/generated/xarray.Dataset.interp.html.

However, there lots of great examples in http://xarray.pydata.org/en/stable/interpolation.html Perhaps I should edit it to: ``` Examples

See http://xarray.pydata.org/en/stable/interpolation.html ```

I was also thinking about adding a note about interpolating time as it came up in https://github.com/pydata/xarray/issues/2284 but not sure what would be the best way to do so.

Code Sample, a copy-pastable example if possible

python import xarray as xr remote_data = xr.open_dataarray('http://iridl.ldeo.columbia.edu/SOURCES/.Models'\ '/.SubX/.RSMAS/.CCSM4/.hindcast/.zg/dods', chunks={'L': 1, 'S': 1}) da = remote_data.isel(P=0,L=0,M=0,X=0,Y=0) da_day_clim = da.groupby('S.dayofyear').mean('S') da_day_clim2 = da_day_clim.chunk({'dayofyear': 366}) da_day_clim_smooth = da_day_clim2.rolling(dayofyear=31, center=True).mean()

Problem description

Initially discussed on SO: https://stackoverflow.com/questions/50265586/xarray-rolling-mean-of-dask-array-conflicting-sizes-for-data-and-coordinate-in

The rolling operation gives a ValueError: conflicting sizes for dimension 'dayofyear': length 351 on the data but length 366 on coordinate 'dayofyear' The length of 351 in the data is created in the rolling operation.

Here's the full traceback: ``` ValueError Traceback (most recent call last) <ipython-input-57-6acf382cdd3d> in <module>() 4 da_day_clim = da.groupby('S.dayofyear').mean('S') 5 da_day_clim2 = da_day_clim.chunk({'dayofyear': 366}) ----> 6 da_day_clim_smooth = da_day_clim2.rolling(dayofyear=31, center=True).mean()

~/anaconda/envs/SubXNAO/lib/python3.6/site-packages/xarray/core/rolling.py in wrapped_func(self, **kwargs) 307 if self.center: 308 values = values[valid] --> 309 result = DataArray(values, self.obj.coords) 310 311 return result

~/anaconda/envs/SubXNAO/lib/python3.6/site-packages/xarray/core/dataarray.py in init(self, data, coords, dims, name, attrs, encoding, fastpath) 224 225 data = as_compatible_data(data) --> 226 coords, dims = _infer_coords_and_dims(data.shape, coords, dims) 227 variable = Variable(dims, data, attrs, encoding, fastpath=True) 228

~/anaconda/envs/SubXNAO/lib/python3.6/site-packages/xarray/core/dataarray.py in _infer_coords_and_dims(shape, coords, dims) 79 raise ValueError('conflicting sizes for dimension %r: ' 80 'length %s on the data but length %s on ' ---> 81 'coordinate %r' % (d, sizes[d], s, k)) 82 83 if k in sizes and v.shape != (sizes[k],):

ValueError: conflicting sizes for dimension 'dayofyear': length 351 on the data but length 366 on coordinate 'dayofyear' ```

Expected Output

The rolling operation would work on the dask array as it would on the dataarray e.g. ``` import pandas as pd import xarray as xr import numpy as np

dates = pd.date_range('1/1/1980', '31/12/2000', freq='D') data = np.linspace(1, len(dates), num=len(dates), dtype=np.float) da = xr.DataArray(data, coords=[dates], dims='time') da_day_clim = da.groupby('time.dayofyear').mean('time') da_day_clim_smooth = da_day_clim.rolling(dayofyear=31, center=True).mean() ```

Output of xr.show_versions()

Code Sample, a copy-pastable example if possible

```python da = xr.DataArray(np.array([1,2,3,4], dtype=np.float).reshape(2,2), ... coords=[pd.date_range('1/1/2000', '1/2/2000', freq='D'), ... np.linspace(0,1,num=2)], ... dims=['time', 'latitude'])

da.resample(freq='M', dim='time', how='mean')

/Users/Ray/anaconda/envs/rot-eof-dev-env/bin/ipython:1: DeprecationWarning:

.resample() has been modified to defer calculations. Instead of passing 'dim' and 'how="mean", #instead consider using .resample(time="M").mean()

#!/Users/Ray/anaconda/envs/rot-eof-dev-env/bin/python

Out[66]:

<xarray.DataArray (time: 1, latitude: 2)>

array([[2., 3.]])

Coordinates:

* time (time) datetime64[ns] 2000-01-31

* latitude (latitude) float64 0.0 1.0

da.resample(time="M").mean()

<xarray.DataArray (time: 1)>

array([2.5])

Coordinates:

* time (time) datetime64[ns] 2000-01-31

```

Problem description

The DeprecationWarning example seems to only work for 1d arrays as it doesn't average along any dimension.

A quick fix could be to show the warning only if the DataArray/Dataset is 1D.

A more thorough fix could be to wrap .resample(time="M").mean() as .resample(freq='M', dim='time', how='mean')???

Expected Output

Same as da.resample(freq='M', dim='time', how='mean')

Output of xr.show_versions()

• [NA] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [NA] Tests added (for all bug fixes or enhancements)
• [NA] Tests passed (for all non-documentation changes)
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Added a resample example to reflect the DeprecationWarning: .resample() has been modified to defer calculations. Instead of passing 'dim' and 'how="mean", instead consider using.... There were also some missing parameters in the docs.

Made a minor edit to my rolling example which uses the parameter center. I can't remember if that parameter was there last time but I think it's useful to return the time value for the middle of window.

Added a groupby example. Not sure if this should go here. See text below for my motivation of putting an example in the docstring of the class/object.

I learnt xarray before I learnt pandas so these example will hopefully be useful to other beginners as it took me a minute to get the syntax right.

Whilst there are great examples of these functions in the docs, a google search of a function for example xarray rolling often returns the doc entry that that object. Having a little example there is helpful for me to get the syntax right. The key/value pair in a dictionary for example is very powerful and always takes me a few tries to get the entry right.

• [NA] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [NA] Tests added (for all bug fixes or enhancements)
• [NA] Tests passed (for all non-documentation changes)
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

This seems to be happening to a lot of projects as highlighted on the recent list of people who have cited this issue https://github.com/badges/shields/issues/716.

Here is a project with a recent commit to resolve this which explains the issue better that I can https://github.com/landscapeio/pylint-celery/pull/7

• [NA] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [NA] Tests added (for all bug fixes or enhancements)
• [NA] Tests passed (for all non-documentation changes)
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

These were inspired by a discussion on the mailing list: https://groups.google.com/forum/#!topic/xarray/FyCFkuWbFSk

• [NA] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [NA] Tests added (for all bug fixes or enhancements)
• [NA] Tests passed (for all non-documentation changes)
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

Closes #1757

I noticed the open_dataset docs provides the full input parameters whereas, open_dataarray docs still lists *args, **kwargs If you point me to where this is created I can change it, if the change is preferred

• [NA] Closes #xxxx (remove if there is no corresponding issue, which should only be the case for minor changes)
• [NA] Tests added (for all bug fixes or enhancements)
• [NA] Tests passed (for all non-documentation changes)
• [NA] Passes git diff upstream/master **/*py | flake8 --diff (remove if you did not edit any Python files)
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API (remove if this change should not be visible to users, e.g., if it is an internal clean-up, or if this is part of a larger project that will be documented later)

• [NA] Closes #xxxx
• [NA] Tests added / passed
• [ ] Passes git diff upstream/master | flake8 --diff
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API

I added an example to apply as I just learnt about it from my Q on stackoverflow (https://stackoverflow.com/questions/46980823/python-xarray-compute-the-absolute-values-of-a-dataset/46981027#46981027).

Also my first time using git and making edits on the command line rather than the web browser.

This idea for this PR was discussed here: https://stackoverflow.com/questions/45992983/python-xarray-rolling-mean-example

I wasn't able to drop the nans using min_periods so I used the dropna method.

If there is a more eloquent method to my example feel free to point me to it.

• [NA] Closes #xxxx
• [NA] Tests added / passed
• [ ] Passes git diff upstream/master | flake8 --diff
• [NA] Fully documented, including whats-new.rst for all changes and api.rst for new API