We tried several times with 2000MB this configuration in the thredds: <Opendap> <ascLimit>50</ascLimit> <binLimit>2000</binLimit> <serverVersion>opendap/3.7</serverVersion> </Opendap> But when we request more than a chunk of time=500MB the error appears: RuntimeError: NetCDF: Access failure

You might want to experiment with smaller chunks.

I tried with 50MB and the elapsed time was huge.

Local Network - Elapsed time: 0.5819 minutes OpenDAP - Elapsed time: 37.1448 minutes

I followed your recommendations @rabernat, please see my test code bellow.

```python import xarray as xr import os from datetime import datetime, timedelta import pandas as pd import shutil import numpy as np import time

lonlat_box = [-4.5, -2.5, 44, 45]

ERA5 IHdata - Local

-------------------

ds = xr.open_mfdataset(['raw/Wind_ERA5_Global_1998.05.nc', 'raw/Wind_ERA5_Global_1998.06.nc']) ds = ds.get('u')

from 0º,360º to -180º,180º

ds['lon'] = (ds.lon + 180) % 360 - 180

lat is upside down --> sort ascending

ds = ds.sortby(['lon', 'lat'])

Make the selection

ds = ds.sel(lon=slice(lonlat_box[0], lonlat_box[1]), lat=slice(lonlat_box[2], lonlat_box[3]))

print(ds)

tic = time.perf_counter() df = ds.to_dataframe() toc = time.perf_counter() print(f"\nLocal Network - Elapsed time: {(toc - tic)/60:0.4f} minutes\n\n")

del ds, df

ERA5 IHdata - Opendap

---------------------

ds = xr.open_mfdataset(['http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.05.nc', 'http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.06.nc'], chunks={'time': '500MB'})

ds = ds.get('u')

from 0º,360º to -180º,180º

ds['lon'] = (ds.lon + 180) % 360 - 180

lat is upside down --> sort ascending

ds = ds.sortby(['lon', 'lat'])

Make the selection

ds = ds.sel(lon=slice(lonlat_box[0], lonlat_box[1]), lat=slice(lonlat_box[2], lonlat_box[3]))

print(ds)

tic = time.perf_counter() df = ds.to_dataframe() toc = time.perf_counter() print(f"\n OpenDAP - Elapsed time: {(toc - tic)/60:0.4f} minutes\n\n")

del ds, df Result:ipython <xarray.DataArray 'u' (lat: 5, lon: 9, time: 1464)> dask.array<getitem, shape=(5, 9, 1464), dtype=float32, chunksize=(5, 9, 744), chunktype=numpy.ndarray> Coordinates: * lon (lon) float32 -4.5 -4.25 -4.0 -3.75 -3.5 -3.25 -3.0 -2.75 -2.5 * lat (lat) float32 44.0 44.25 44.5 44.75 45.0 * time (time) datetime64[ns] 1998-05-01 ... 1998-06-30T23:00:00 Attributes: units: m s**-1 long_name: 10 metre U wind component

Local Network - Elapsed time: 0.4037 minutes

<xarray.DataArray 'u' (lat: 5, lon: 9, time: 1464)> dask.array<getitem, shape=(5, 9, 1464), dtype=float32, chunksize=(5, 9, 120), chunktype=numpy.ndarray> Coordinates: * lon (lon) float32 -4.5 -4.25 -4.0 -3.75 -3.5 -3.25 -3.0 -2.75 -2.5 * lat (lat) float32 44.0 44.25 44.5 44.75 45.0 * time (time) datetime64[ns] 1998-05-01 ... 1998-06-30T23:00:00 Attributes: units: m s**-1 long_name: 10 metre U wind component

OpenDAP - Elapsed time: 8.1971 minutes

```

Using this chunk of time=500Mb the code runs properly but it is really slow compared with the response through local network. I will try to raise this limit in the Opendap configuration with our IT-team to a more reasonable limit.

@rabernat - Thank you! I will review the code (thank you for the extra comments, I really appreciate that) and follow your instructions to test the chunk size.

Just for my understanding, So theoretically It is not possible to make big requests without using chunking? The threads server is under our management and we want to know if these errors can be solved through any specific configuration of the service in the thredds.

Thank you in advance!

thank you @dcherian, We know that if the request is small it works fine, but we want to make big requests of data. Is any limitation using opendap?

Thank you @ocefpaf!

But it raised the same error. I also try to load "u" variable with matlab ncread through opendap and also failed! So maybe is not a problem related with python...? I am very confused!

```Loading files: http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.05.nc http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.06.nc

RuntimeError Traceback (most recent call last) d:\2020_REPSOL\Codigos_input_TESEO\user_script.py in 58 # ) 59 ---> 60 ERA5_windIHData2txt_TESEO(lonlat_box=[-4.5, -2.5, 44, 45], 61 date_ini=datetime(1998, 5, 28, 0), 62 date_end=datetime(1998, 6, 1, 12),

d:\2020_REPSOL\Codigos_input_TESEO\TESEOtools_v0.py in ERA5_windIHData2txt_TESEO(failed resolving arguments) 826 827 # From xarray to dataframe --> 828 df = ds.to_dataframe().reset_index() 829 del ds 830 print('[Processing currents 2D...]')

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\dataset.py in to_dataframe(self) 4503 this dataset's indices. 4504 """ -> 4505 return self._to_dataframe(self.dims) 4506 4507 def _set_sparse_data_from_dataframe(

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\dataset.py in _to_dataframe(self, ordered_dims) 4489 def _to_dataframe(self, ordered_dims): 4490 columns = [k for k in self.variables if k not in self.dims] -> 4491 data = [ 4492 self._variables[k].set_dims(ordered_dims).values.reshape(-1) 4493 for k in columns

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\dataset.py in (.0) 4490 columns = [k for k in self.variables if k not in self.dims] 4491 data = [ -> 4492 self._variables[k].set_dims(ordered_dims).values.reshape(-1) 4493 for k in columns 4494 ]

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\variable.py in values(self) 444 def values(self): 445 """The variable's data as a numpy.ndarray""" --> 446 return _as_array_or_item(self._data) 447 448 @values.setter

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\variable.py in _as_array_or_item(data) 247 TODO: remove this (replace with np.asarray) once these issues are fixed 248 """ --> 249 data = np.asarray(data) 250 if data.ndim == 0: 251 if data.dtype.kind == "M":

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\numpy\core_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\array\core.py in array(self, dtype, kwargs) 1334 1335 def array(self, dtype=None, kwargs): -> 1336 x = self.compute() 1337 if dtype and x.dtype != dtype: 1338 x = x.astype(dtype)

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\base.py in compute(self, kwargs) 164 dask.base.compute 165 """ --> 166 (result,) = compute(self, traverse=False, kwargs) 167 return result 168

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\base.py in compute(args, kwargs) 442 postcomputes.append(x.dask_postcompute()) 443 --> 444 results = schedule(dsk, keys, kwargs) 445 return repack([f(r, a) for r, (f, a) in zip(results, postcomputes)]) 446

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\threaded.py in get(dsk, result, cache, num_workers, pool, **kwargs) 74 pools[thread][num_workers] = pool 75 ---> 76 results = get_async( 77 pool.apply_async, 78 len(pool._pool),

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs) 484 _execute_task(task, data) # Re-execute locally 485 else: --> 486 raise_exception(exc, tb) 487 res, worker_id = loads(res_info) 488 state["cache"][key] = res

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\local.py in reraise(exc, tb) 314 if exc.traceback is not tb: 315 raise exc.with_traceback(tb) --> 316 raise exc 317 318

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\local.py in execute_task(key, task_info, dumps, loads, get_id, pack_exception) 220 try: 221 task, data = loads(task_info) --> 222 result = _execute_task(task, data) 223 id = get_id() 224 result = dumps((result, id))

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in _execute_task(arg, cache, dsk) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in (.0) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in _execute_task(arg, cache, dsk) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in (.0) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in _execute_task(arg, cache, dsk) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in (.0) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\core.py in _execute_task(arg, cache, dsk) 119 # temporaries by their reference count and can execute certain 120 # operations in-place. --> 121 return func(*(_execute_task(a, cache) for a in args)) 122 elif not ishashable(arg): 123 return arg

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\dask\array\core.py in getter(a, b, asarray, lock) 98 c = a[b] 99 if asarray: --> 100 c = np.asarray(c) 101 finally: 102 if lock:

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\numpy\core_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\indexing.py in array(self, dtype) 489 490 def array(self, dtype=None): --> 491 return np.asarray(self.array, dtype=dtype) 492 493 def getitem(self, key):

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\numpy\core_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\indexing.py in array(self, dtype) 651 652 def array(self, dtype=None): --> 653 return np.asarray(self.array, dtype=dtype) 654 655 def getitem(self, key):

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\numpy\core_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\indexing.py in array(self, dtype) 555 def array(self, dtype=None): 556 array = as_indexable(self.array) --> 557 return np.asarray(array[self.key], dtype=None) 558 559 def transpose(self, order):

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\numpy\core_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\coding\variables.py in array(self, dtype) 70 71 def array(self, dtype=None): ---> 72 return self.func(self.array) 73 74 def repr(self):

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\coding\variables.py in _scale_offset_decoding(data, scale_factor, add_offset, dtype) 216 217 def _scale_offset_decoding(data, scale_factor, add_offset, dtype): --> 218 data = np.array(data, dtype=dtype, copy=True) 219 if scale_factor is not None: 220 data *= scale_factor

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\coding\variables.py in array(self, dtype) 70 71 def array(self, dtype=None): ---> 72 return self.func(self.array) 73 74 def repr(self):

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\coding\variables.py in _apply_mask(data, encoded_fill_values, decoded_fill_value, dtype) 136 ) -> np.ndarray: 137 """Mask all matching values in a NumPy arrays.""" --> 138 data = np.asarray(data, dtype=dtype) 139 condition = False 140 for fv in encoded_fill_values:

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\numpy\core_asarray.py in asarray(a, dtype, order) 83 84 """ ---> 85 return array(a, dtype, copy=False, order=order) 86 87

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\indexing.py in array(self, dtype) 555 def array(self, dtype=None): 556 array = as_indexable(self.array) --> 557 return np.asarray(array[self.key], dtype=None) 558 559 def transpose(self, order):

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\backends\netCDF4_.py in getitem(self, key) 70 71 def getitem(self, key): ---> 72 return indexing.explicit_indexing_adapter( 73 key, self.shape, indexing.IndexingSupport.OUTER, self._getitem 74 )

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\core\indexing.py in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method) 835 """ 836 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support) --> 837 result = raw_indexing_method(raw_key.tuple) 838 if numpy_indices.tuple: 839 # index the loaded np.ndarray

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\backends\netCDF4_.py in _getitem(self, key) 83 with self.datastore.lock: 84 original_array = self.get_array(needs_lock=False) ---> 85 array = getitem(original_array, key) 86 except IndexError: 87 # Catch IndexError in netCDF4 and return a more informative

~\AppData\Local\Continuum\miniconda3\envs\TEST\lib\site-packages\xarray\backends\common.py in robust_getitem(array, key, catch, max_retries, initial_delay) 52 for n in range(max_retries + 1): 53 try: ---> 54 return array[key] 55 except catch: 56 if n == max_retries:

netCDF4_netCDF4.pyx in netCDF4._netCDF4.Variable.getitem()

netCDF4_netCDF4.pyx in netCDF4._netCDF4.Variable._get()

netCDF4_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Access failure```

thank you @ocefpaf ,

I installed xarray through the recommended command in the official website in my minicoda env some months-year ago: python conda install -c conda-forge xarray dask netCDF4 bottleneck

I list my versions below: ``` INSTALLED VERSIONS

commit: None python: 3.6.7 (default, Feb 28 2019, 07:28:18) [MSC v.1900 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None libhdf5: 1.10.4 libnetcdf: 4.6.2

xarray: 0.12.1 pandas: 0.24.2 numpy: 1.16.3 scipy: 1.2.1 netCDF4: 1.5.1.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 nc_time_axis: 1.2.0 PseudonetCDF: None rasterio: None cfgrib: 0.9.6.2 iris: None bottleneck: None dask: 1.1.5 distributed: 1.28.1 matplotlib: 3.0.3 cartopy: 0.16.0 seaborn: None setuptools: 41.0.1 pip: 19.1.1 conda: 4.8.2 pytest: None IPython: 7.5.0 sphinx: None I'v just created a new environment with python3.7 and all last versions and the result is the same error, I list this new environment below also: INSTALLED VERSIONS

commit: None python: 3.7.7 (default, May 6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None libhdf5: 1.10.4 libnetcdf: 4.7.3

xarray: 0.15.1 pandas: 1.0.3 numpy: 1.18.1 scipy: 1.4.1 netCDF4: 1.5.3 pydap: installed h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.1.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.3.2 dask: 2.15.0 distributed: 2.15.2 matplotlib: None cartopy: None seaborn: None numbagg: None setuptools: 46.1.3.post20200330 pip: 20.0.2 conda: None pytest: None IPython: 7.13.0 sphinx: None ```

Thank you in advance!

Totally agree,

from my code the list of url are: python Loading files: http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.05.nc http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.06.nc and through the web browser I can copy paste for that dates this:

python http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.05.nc http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.06.nc

So I think the URL is properly constructed, indeed if I select only the longitude variable, which is quit small, I can perform the ds.to_dataframe() method... so I think url is fine!