html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/3666#issuecomment-571691258,https://api.github.com/repos/pydata/xarray/issues/3666,571691258,MDEyOklzc3VlQ29tbWVudDU3MTY5MTI1OA==,2448579,2020-01-07T17:35:51Z,2020-01-07T17:36:03Z,MEMBER,"@tlogan specifying `use_cftime` as True or False in the `open_dataset` call might be useful. This suggests that the `RecursionError` in @keewis example is a result of trying to concatenate datasets with mixed `DatetimeIndex` & `CFTimeIndex` indexes. ``` In [18]: files = glob.glob(""*.nc"") ...: datasets = [xr.open_dataset(file) for file in files] ...: xr.concat(datasets, dim=""new_dim"") RecursionError ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571689678,https://api.github.com/repos/pydata/xarray/issues/3666,571689678,MDEyOklzc3VlQ29tbWVudDU3MTY4OTY3OA==,22454970,2020-01-07T17:31:53Z,2020-01-07T17:31:53Z,NONE,"FYI - @dcherian @keewis ... thanks for the suggestions and help This final small change (dropping time values before replacing with the common calendar) has gotten all cases in the xclim code test suite running against the xarray@master... Feel free to close ````dslist =[] for d in datasets: ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False) cal1 = xr.decode_cf(ds).time ds = ds.drop_vars('time') ds[""time""] = pd.to_datetime( { ""year"": cal1.time.dt.year, ""month"": cal1.time.dt.month, ""day"": cal1.time.dt.day, } ).values dslist.append(ds) ens1 = xr.concat(dslist,dim='realization') #ens1 = ensembles.create_ensemble(datasets) print(ens1)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571669562,https://api.github.com/repos/pydata/xarray/issues/3666,571669562,MDEyOklzc3VlQ29tbWVudDU3MTY2OTU2Mg==,22454970,2020-01-07T16:42:44Z,2020-01-07T16:42:44Z,NONE,"I think I need to leave the ds.time as undecoded and create a calendar variable each time then overwrite the ds values: ```import wget import glob import xarray as xr import pandas as pd # def install(package): # subprocess.check_call([sys.executable, ""-m"", ""pip"", ""install"", package]) # try: # from xclim import ensembles # except: # install('xclim') # from xclim import ensembles outdir = '/home/travis/Downloads' url = [] url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_ACCESS1-0_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_BNU-ESM_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r2i1p1_1950-2100_tg_mean_YS.nc') for u in url: wget.download(u,out=outdir) datasets = glob.glob(f'{outdir}/*1950*.nc') dslist =[] for d in datasets: ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False) cal1 = xr.decode_cf(ds).time ds[""time""].values = pd.to_datetime( { ""year"": cal1.time.dt.year, ""month"": cal1.time.dt.month, ""day"": cal1.time.dt.day, } ) dslist.append(ds) ens1 = xr.concat(dslist,dim='realization') #ens1 = ensembles.create_ensemble(datasets) print(ens1)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571668871,https://api.github.com/repos/pydata/xarray/issues/3666,571668871,MDEyOklzc3VlQ29tbWVudDU3MTY2ODg3MQ==,22454970,2020-01-07T16:41:07Z,2020-01-07T16:41:07Z,NONE,"This is more or less what I was doing but I think the problem may be that I am trying to overwrite the cfdatetime with the pd.datetime in a loop see code below: ```import wget import glob import xarray as xr import pandas as pd # def install(package): # subprocess.check_call([sys.executable, ""-m"", ""pip"", ""install"", package]) # try: # from xclim import ensembles # except: # install('xclim') # from xclim import ensembles outdir = '/home/travis/Downloads' url = [] url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_ACCESS1-0_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_BNU-ESM_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r1i1p1_1950-2100_tg_mean_YS.nc') url.append('https://github.com/Ouranosinc/xclim/raw/master/tests/testdata/EnsembleStats/BCCAQv2+ANUSPLIN300_CCSM4_historical+rcp45_r2i1p1_1950-2100_tg_mean_YS.nc') for u in url: wget.download(u,out=outdir) datasets = glob.glob(f'{outdir}/*1950*.nc') dslist =[] for d in datasets: ds = xr.open_dataset(d, chunks=dict(time=10), decode_times=False) ds['time'] = xr.decode_cf(ds).time ds[""time""].values = pd.to_datetime( { ""year"": ds.time.dt.year, ""month"": ds.time.dt.month, ""day"": ds.time.dt.day, } ) dslist.append(ds) ens1 = xr.concat(dslist,dim='realization') ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571667543,https://api.github.com/repos/pydata/xarray/issues/3666,571667543,MDEyOklzc3VlQ29tbWVudDU3MTY2NzU0Mw==,14808389,2020-01-07T16:38:04Z,2020-01-07T16:38:04Z,MEMBER,"it does. Also, no errors in either case, so the problem is with `DatetimeIndex`, `CFTimeIndex` or the way the data is converted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571665270,https://api.github.com/repos/pydata/xarray/issues/3666,571665270,MDEyOklzc3VlQ29tbWVudDU3MTY2NTI3MA==,2448579,2020-01-07T16:33:06Z,2020-01-07T16:33:06Z,MEMBER,I'm guessing `decode_times=False` will fix that bit.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571641613,https://api.github.com/repos/pydata/xarray/issues/3666,571641613,MDEyOklzc3VlQ29tbWVudDU3MTY0MTYxMw==,22454970,2020-01-07T15:42:37Z,2020-01-07T15:42:37Z,NONE,"Ok thanks, Basically the xclim ensembles code should overwrite a common calendar pd.datetime format to these monthly datasets (various calendar type) before concatenation (as mixing calendars is not possible). I will try to write it as an xarray only example","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571639956,https://api.github.com/repos/pydata/xarray/issues/3666,571639956,MDEyOklzc3VlQ29tbWVudDU3MTYzOTk1Ng==,14808389,2020-01-07T15:38:57Z,2020-01-07T15:38:57Z,MEMBER,"I can reproduce this error with your files: ```python In [18]: files = glob.glob(""*.nc"") ...: datasets = [xr.open_dataset(file) for file in files] ...: xr.concat(datasets, dim=""new_dim"") RecursionError ``` but sorting the files before concatenating results in a different error: ```python In [19]: files = glob.glob(""*.nc"") ...: datasets = [xr.open_dataset(file) for file in sorted(files)] ...: xr.concat(datasets, dim=""new_dim"") TypeError: cannot compare cftime.DatetimeNoLeap(1950-01-01 00:00:00) and Timestamp('1950-01-01 00:00:00') (different calendars) ``` which might mean that there is a problem with the data. Regardless, we might want to make sure the recursion does not happen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413 https://github.com/pydata/xarray/issues/3666#issuecomment-571639751,https://api.github.com/repos/pydata/xarray/issues/3666,571639751,MDEyOklzc3VlQ29tbWVudDU3MTYzOTc1MQ==,2448579,2020-01-07T15:38:28Z,2020-01-07T15:38:28Z,MEMBER,"Thanks @tlogan2000. Can you reduce this example to just a `xarray.concat` call?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,546303413