issue_comments

Closing in favour of https://github.com/pydata/xarray/issues/7369

True.

So it sounds like what you're hoping for is separate per-column or per-row color scaling?

Exactly.

@shoyer could you maybe provide a code example of the current index aligned behaviour and a future not index aligned behaviour?

I am a bit worried about transitioning previous code bases to such new xarray releases

It is possible to set ECCODES_DEFINITION_PATH in eccodes, see http://download.ecmwf.int/test-data/eccodes/html/namespaces.html. But the codetab you want to use here from MPIESM grb output is a CDO parameter table, see https://code.mpimet.mpg.de/projects/cdo/embedded/cdo.pdf#section.1.8, so in my understanding something CDO internal and likely not usable with eccodes/cfgrib.

To be implemented here https://github.com/pydata/xarray/blob/d479009d79374dc4a56c9f4346b1af38f5ac182c/xarray/backends/plugins.py#L102

Nice work @aaronspring! I think this is ready to go.

Thank you so much @spencerkclark for your guidance

thank you very much for the guidance here. I hope this is now close to how you envisioned this. I merely wanted to get a small improvement in functionality and didnt expect to learn so much. thank you so far @spencerkclark

failing tests are unrelated to cftime IMO

In my understanding we can get rid of fkirc/skip-duplicate-actions and cancel-duplicate-runs.yaml

pre-commit.ci autofix

thanks for the reviews so far @spencerkclark @Illviljan @max-sixty - I think I implemented the suggestions and am looking forward to your next reviews

Would you remove the isinstance checks? I though that mypy does check at runtime, so it would detect a wrong input by the user. (I am still new to typing etc)

code in xarray is nearly identical to pandas. Maybe I misunderstood the documentation/explanation, i.e. meaning of month in freq-month for seasonal

the same happens also in pandas... ```python import pandas as pd i_pd = pd.date_range(start="2000-01-01", end="2002-01-01", freq="6MS") i_pd DatetimeIndex(['2000-01-01', '2000-07-01', '2001-01-01', '2001-07-01', '2002-01-01'], dtype='datetime64[ns]', freq='6MS')

pd.infer_freq(i_pd) '2QS-OCT' ```

bokeh etc not installed anymore when temporarily not installing iris from conda https://github.com/pydata/xarray/runs/4298565586?check_suite_focus=true

This makes sense, but it sounds like this suggestion (of accepting Datasets not just DataArrays) is mostly a convenience tool for applying histograms to particular variables across multiple datasets quickly. It's not fundamentally different to picking and choosing the variables you want from multiple datasets and feeding them in to histogram as dataarrays.

agree.

I think we should focus on including features that enable analyses that would otherwise be difficult or impossible, for example ND bins

looking forward to the PR

I like your explanation of the two different inputs @dougiesquire and for multi-dim datasets these must be xr.datasets. my point about the bins is that if the inputs are two xr.datasets, then also the bins should be two xr.datasets. If the bins were only two xr.Arrays, then how could these bins discriminate for the different variables in the inputs?

I am unsure about this and cannot manage to put my Südasien down precisely.

Calculating a contingency table for instance between two multivar inputs:

ˋˋˋ xhistogram(ds_observations_multivar, ds_forecast_multivar, bins=[ds_obs_multivar_edges, ds_forecast_multivar_edges ]) ˋˋˋ

as in https://github.com/xarray-contrib/xskillscore/blob/6f7be06098eefa1cdb90f7319f577c274621301c/xskillscore/core/contingency.py#L156

maybe @dougiesquire can phrase this more precisely

What I'm unclear about is what you want to achieve by inputting an xarray.Dataset that couldn't be done with inputs of ND xr.DataArrays as both data and bins?

with dataset bins I want to have different bin_edges for each dataset. If bins is only a dataArray, I cannot have this. Can I?

I tried to show in https://gist.github.com/aaronspring/251553f132202cc91aadde03f2a452f9 how I would like to use xr.Datasets as bins, e.g. defining bin edges based on quantiles of the climatology, i.e. bin edges depend on lon, lat, maybe weekofyear and variable.

I tried show in the gist that I could be also nice to allow xr.Datasets as bins if the inputs are xr.Datasets.

You can already choose different bins to use for each input variable`

I cannot find this in #5400. I should checkout and run the code locally.

Yep, the example xskillscore code posted doesnt allow nd bins. forgot that. correct. in my head thinking about the future it does. https://github.com/xarray-contrib/xskillscore/blob/6f7be06098eefa1cdb90f7319f577c274621301c/xskillscore/core/probabilistic.py#L498 takes xr.Datasets as bins and in a previous version we used xhist but then changed to make this run on nd arrays and xr.Datasets.

what about a list of xarray.Datasets as bins? suppose you have an xr.Dataset you want to bin different variables for different bins (eg from xr.Dataset.quantile)

@dougiesquire implemented this in https://github.com/xarray-contrib/xskillscore/blob/2217b58c536ec1b3d2c42265ed6689a740c2b3bf/xskillscore/core/utils.py#L133

EDIT: seeing now that this issue and #5400 aims to implement xr.DataArray.hist only. xr.Dataset would be also nice :)

1. exists as ds.time.to_index().calendar
2. would like to use

Thanks for the suggestion with xr.align.

my speculation is that xs.pearson_r is a bit faster because we first write the whole function in numpy and then pass it through xr.apply_ufunc. I think therefore it only works for xr but not dask.da

Your function from the notebook could also easily implement the builtin weighted function

We implemented xr.corr as xr.pearson_r in https://xskillscore.readthedocs.io/en/stable/api/xskillscore.pearson_r.html#xskillscore.pearson_r and it’s ~30% faster than xr.corr see #4768

Thanks for the smooth review process @spencerkclark @keewis

Ah I need to account for the docstring examples with only one or two items, maybe dropping freq then

also should it be frequency or freq?

I also get this error locally. somehow in some envs xarray (but not in others) tries to use daysinmonth. I also know about days_in_month, but just changing that in frequencies.py https://github.com/pydata/xarray/blob/d9ebcafaddf0bb80052d30f43ca5ea18a5c7229f/xarray/coding/frequencies.py#L261 didnt help it.

``` ____________ test_cftimeindex_freq_in_repr[MS-noleap] _______________

freq = 'MS', calendar = 'noleap'

@requires_cftime
@pytest.mark.parametrize("calendar", ["noleap", "360_day"])
@pytest.mark.parametrize("freq", ["1D", "MS"])
def test_cftimeindex_freq_in_repr(freq,calendar):
    index = xr.cftime_range(start="2000", periods=3, freq=freq, calendar=calendar)

  assert f', freq={freq}' in index.__repr__()

/Users/aaron.spring/Coding/xarray/xarray/tests/test_cftimeindex.py:952:

/Users/aaron.spring/Coding/xarray/xarray/coding/cftimeindex.py:342: in repr attrs_str = format_attrs(self) /Users/aaron.spring/Coding/xarray/xarray/coding/cftimeindex.py:263: in format_attrs "freq": f"'{index.freq}'" /Users/aaron.spring/Coding/xarray/xarray/coding/cftimeindex.py:691: in freq return infer_freq(self) /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:97: in infer_freq return inferer.get_freq() /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:132: in get_freq return self._infer_daily_rule() /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:165: in _infer_daily_rule monthly_rule = self._get_monthly_rule() /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:199: in _get_monthly_rule return {"cs": "MS", "ce": "M"}.get(month_anchor_check(self.index))

dates = <[AttributeError("'cftime._cftime.DatetimeNoLeap' object has no attribute 'daysinmonth'",) raised in repr()] CFTimeIndex object at 0x7fca9312b160>

def month_anchor_check(dates):
    """Return the monthly offset string.

    Return "cs" if all dates are the first days of the month,
    "ce" if all dates are the last day of the month,
    None otherwise.

    Replicated pandas._libs.tslibs.resolution.month_position_check
    but without business offset handling.
    """
    calendar_end = True
    calendar_start = True

    for date in dates:
        if calendar_start:
            calendar_start &= date.day == 1

        if calendar_end:

          cal = date.day == date.daysinmonth

E AttributeError: 'cftime._cftime.DatetimeNoLeap' object has no attribute 'daysinmonth'

/Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:261: AttributeError ____________ test_cftimeindex_freq_in_repr[MS-360_day] ______________

freq = 'MS', calendar = '360_day'

@requires_cftime
@pytest.mark.parametrize("calendar", ["noleap", "360_day"])
@pytest.mark.parametrize("freq", ["1D", "MS"])
def test_cftimeindex_freq_in_repr(freq,calendar):
    index = xr.cftime_range(start="2000", periods=3, freq=freq, calendar=calendar)

  assert f', freq={freq}' in index.__repr__()

/Users/aaron.spring/Coding/xarray/xarray/tests/test_cftimeindex.py:952:

/Users/aaron.spring/Coding/xarray/xarray/coding/cftimeindex.py:342: in repr attrs_str = format_attrs(self) /Users/aaron.spring/Coding/xarray/xarray/coding/cftimeindex.py:263: in format_attrs "freq": f"'{index.freq}'" /Users/aaron.spring/Coding/xarray/xarray/coding/cftimeindex.py:691: in freq return infer_freq(self) /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:97: in infer_freq return inferer.get_freq() /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:132: in get_freq return self._infer_daily_rule() /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:165: in _infer_daily_rule monthly_rule = self._get_monthly_rule() /Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:199: in _get_monthly_rule return {"cs": "MS", "ce": "M"}.get(month_anchor_check(self.index))

dates = <[AttributeError("'cftime._cftime.Datetime360Day' object has no attribute 'daysinmonth'",) raised in repr()] CFTimeIndex object at 0x7fca930cfe10>

def month_anchor_check(dates):
    """Return the monthly offset string.

    Return "cs" if all dates are the first days of the month,
    "ce" if all dates are the last day of the month,
    None otherwise.

    Replicated pandas._libs.tslibs.resolution.month_position_check
    but without business offset handling.
    """
    calendar_end = True
    calendar_start = True

    for date in dates:
        if calendar_start:
            calendar_start &= date.day == 1

        if calendar_end:

          cal = date.day == date.daysinmonth

E AttributeError: 'cftime._cftime.Datetime360Day' object has no attribute 'daysinmonth'

/Users/aaron.spring/Coding/xarray/xarray/coding/frequencies.py:261: AttributeError ===================================================================================== short test summary info ===================================================================================== FAILED xarray/tests/test_cftimeindex.py::test_cftimeindex_freq_in_repr[1D-noleap] - AttributeError: 'cftime._cftime.DatetimeNoLeap' object has no attribute 'daysinmonth' FAILED xarray/tests/test_cftimeindex.py::test_cftimeindex_freq_in_repr[1D-360_day] - AttributeError: 'cftime._cftime.Datetime360Day' object has no attribute 'daysinmonth' FAILED xarray/tests/test_cftimeindex.py::test_cftimeindex_freq_in_repr[MS-noleap] - AttributeError: 'cftime._cftime.DatetimeNoLeap' object has no attribute 'daysinmonth' FAILED xarray/tests/test_cftimeindex.py::test_cftimeindex_freq_in_repr[MS-360_day] - AttributeError: 'cftime._cftime.Datetime360Day' object has no attribute 'daysinmonth' ======================================================================================== 4 failed in 2.58s =========================================================================== ```

using this feature now alot. @spencerkclark what about implementing also freq into the CFTimeIndex repr?

Its hard to see the frequency by eye sometimes, I would just rely on xr.infer_freq and add to the repr str, also if freq is None/indistinguishable

cross posted: https://github.com/Unidata/netcdf4-python/issues/1043

For those who think bool(xr.Dataset(dict(x=False))) should evaluate to False, what do you think dict(x=False) should evaluate to?

good point

IIUC, this is somewhat of an edge case — it's only possible to coerce DataArray to bool when there's exactly one element. (Not that we shouldn't resolve)

I now understand the underlying problem better with your comments. My expectations were too naive about it. I cannot evaluate the consequences of changing this. I close this issue and hopefully users having this problem in the future will find this issue.

currently

python if (xrobject > 10).any(): raise ValueError('blabla') depends on whether xrobject is xr.DataArray or xr.Dataset.

thanks for the explanation. but from a naive user perspective,

python bool(xr.DataArray(False).to_dataset(name='var')) # False bool(xr.DataArray(True).to_dataset(name='var')) # True bool(xr.DataArray(False)) # False bool(xr.DataArray(True)) # True would be nice.

My pleasure. Learnt a lot.

And it was a long-standing issue that was mentioned in a medium article about xarray looking for more contributors.

Thanks @aaronspring; this is looking very close! Just a few more minor suggestions.

I hope this is the final one. all tests pass. implemented your suggestions. took me a few commits, but I learned a lot. thanks for the guidance @spencerkclark

dont understand why isort fails

now aligns to display_width:

python for dw in [40,60,80,120]: with xr.set_options(display_width=dw): print(time[:2],dw,'\n')

``` CFTimeIndex([2000-01-01 00:00:00, 2000-01-02 00:00:00], dtype='object', length=2, calendar='gregorian') 40

CFTimeIndex([2000-01-01 00:00:00, 2000-01-02 00:00:00], dtype='object', length=2, calendar='gregorian') 60

CFTimeIndex([2000-01-01 00:00:00, 2000-01-02 00:00:00], dtype='object', length=2, calendar='gregorian') 80

CFTimeIndex([2000-01-01 00:00:00, 2000-01-02 00:00:00], dtype='object', length=2, calendar='gregorian') 120 ```

@spencerkclark I reimplemented the repr. Mostly went after the visuals. So now this looks like before but including calendar in the repr. Any more tests needed? Would you please give this a look?

I now reimplemented the visuals of pd.Index and also test against pd.Index.repr @spencerkclark

now after clicking on the data item:

this PR is now ready for review. I don't understand why azure tests are failing.

I found a workaround with to_index() in formatting.short_data_repr. this is makes one test fail: xarray/tests/test_dask.py TestDataArrayAndDataset.test_dataarray_repr. is this ok or not a valid way to implement a cftimeindex.repr?

EDIT: I ensure now that to_index is only done when if CFTimeIndex.

Ready for review.

Currently only the cftimeindex repr shows the calendar property. I aim to get it into the dataset/dataarray repr.

would it be a good idea to modify formatting.py.array_repr(array)? IndexVariable uses this repr. Once time is a coordinate, it becomes an IndexVariable. but the index variable doesnt have this cftime.calender property anymore. Should I try to add this calendar property to the array or would this not be desired?

I do understand the meaning of this last sentence @spencerkclark

My sense is that it is more important that indexes with the same date type have the same calendar attribute than that the calendar argument passed to cftime_range is propagated verbatim to the index it produces. What do you think?

Should I rather test on cftimeindex created in a different way?

This doesnt change the html repr yet: when clicking on the data symbol, calendar is not shown in the printout. I will further try to fix this

I was hoping to inherit from pandas like: python def __repr__(self): super().__repr__() return self.__repr__().strip(")")+f", calendar={self.calendar}')"

But I will now try to rebuild as in pandas.

how should I extend the __repr__ coming from pd.Index

if you define a __repr__ method under CFTimeIndex, won't it override pd.Index.__repr__?

Yes. I will do that. My question was whether should replicate the pd.Index.repr or try to import or inherit as much as I can from pandas?

also works: da.data.visualize()

me too on Firefox linux

this is great. thanks for the hint.

would this be a starting point: https://github.com/pydata/xarray/blob/603b0ad3f8a02a9e1180eb8dfc72f7f885f0e19a/xarray/core/variable.py#L2222 ? but I dont see where calendar is coming from.

such a calendar attribute would be nice. Didnt find a workaround. My goal is to get a str for calendar type from ds:

Once implemented, this could then be accessed via ds.time.to_index().calendar. with some help where to start I would go for this.

This was my workaround. https://github.com/aaronspring/pymistral/issues/19

I would use it with the echam6 and jsbach ‘.codes’ files in MPIESM.

I would love that feature. Also found that pyngl docu but didn’t get it working...

I am unsure whether I added enough tests. only one test that checks for different values with or without skipna.

Can the stats functions from https://esmlab.readthedocs.io/en/latest/api.html#statistics-functions be used?

thanks @dcherian that was easier than expected. it speeds up a bit but not much. thanks