@tlogan specifying use_cftime as True or False in the open_dataset call might be useful.

This suggests that the RecursionError in @keewis example is a result of trying to concatenate datasets with mixed DatetimeIndex & CFTimeIndex indexes. In [18]: files = glob.glob("*.nc") ...: datasets = [xr.open_dataset(file) for file in files] ...: xr.concat(datasets, dim="new_dim") RecursionError

FYI - @dcherian @keewis ... thanks for the suggestions and help This final small change (dropping time values before replacing with the common calendar) has gotten all cases in the xclim code test suite running against the xarray@master... Feel free to close

````dslist =[] for d in datasets: ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False) cal1 = xr.decode_cf(ds).time ds = ds.drop_vars('time') ds["time"] = pd.to_datetime( { "year": cal1.time.dt.year, "month": cal1.time.dt.month, "day": cal1.time.dt.day, } ).values dslist.append(ds)

ens1 = xr.concat(dslist,dim='realization')

ens1 = ensembles.create_ensemble(datasets)

print(ens1)

I think I need to leave the ds.time as undecoded and create a calendar variable each time then overwrite the ds values: ```import wget import glob import xarray as xr import pandas as pd

def install(package):

subprocess.check_call([sys.executable, "-m", "pip", "install", package])

try:

from xclim import ensembles

except:

install('xclim')

from xclim import ensembles

outdir = '/home/travis/Downloads' url = [] url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_ACCESS1-0_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_BNU-ESM_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r2i1p1_1950-2100_tg_mean_YS.nc') for u in url: wget.download(u,out=outdir) datasets = glob.glob(f'{outdir}/1950.nc')

dslist =[] for d in datasets: ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False) cal1 = xr.decode_cf(ds).time ds["time"].values = pd.to_datetime( { "year": cal1.time.dt.year, "month": cal1.time.dt.month, "day": cal1.time.dt.day, } ) dslist.append(ds)

ens1 = xr.concat(dslist,dim='realization')

ens1 = ensembles.create_ensemble(datasets)

print(ens1)

This is more or less what I was doing but I think the problem may be that I am trying to overwrite the cfdatetime with the pd.datetime in a loop see code below:

```import wget import glob import xarray as xr import pandas as pd

def install(package):

subprocess.check_call([sys.executable, "-m", "pip", "install", package])

try:

from xclim import ensembles

except:

install('xclim')

from xclim import ensembles

outdir = '/home/travis/Downloads' url = [] url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_ACCESS1-0_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_BNU-ESM_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r2i1p1_1950-2100_tg_mean_YS.nc') for u in url: wget.download(u,out=outdir) datasets = glob.glob(f'{outdir}/1950.nc')

dslist =[] for d in datasets: ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False) ds['time'] = xr.decode_cf(ds).time ds["time"].values = pd.to_datetime( { "year": ds.time.dt.year, "month": ds.time.dt.month, "day": ds.time.dt.day, } ) dslist.append(ds)

ens1 = xr.concat(dslist,dim='realization')

it does. Also, no errors in either case, so the problem is with DatetimeIndex, CFTimeIndex or the way the data is converted.

I'm guessing decode_times=False will fix that bit.

Ok thanks, Basically the xclim ensembles code should overwrite a common calendar pd.datetime format to these monthly datasets (various calendar type) before concatenation (as mixing calendars is not possible). I will try to write it as an xarray only example

I can reproduce this error with your files: python In [18]: files = glob.glob("*.nc") ...: datasets = [xr.open_dataset(file) for file in files] ...: xr.concat(datasets, dim="new_dim") RecursionError but sorting the files before concatenating results in a different error: python In [19]: files = glob.glob("*.nc") ...: datasets = [xr.open_dataset(file) for file in sorted(files)] ...: xr.concat(datasets, dim="new_dim") TypeError: cannot compare cftime.DatetimeNoLeap(1950-01-01 00:00:00) and Timestamp('1950-01-01 00:00:00') (different calendars) which might mean that there is a problem with the data.

Regardless, we might want to make sure the recursion does not happen.

Thanks @tlogan2000.

Can you reduce this example to just a xarray.concat call?