issues

This merge request allows the user to set a backend_env while opening a file. This should be a dictionary, and the key/value pairs are temporarily added to os.enviorn while opening the file. The old environment is restored later.

• [x] Closes #3853 Maybe -- I'm not sure if it is clever to actually close this issue at this point. I added the backend_env idea.
• [ ] Tests added

Here, I need some help: How should I actually design the tests? The environment is only temporarily modified, so as soon as the open_dataset function ends again, the environment is restored. I would have though temporarily adding an equivalent to $ export lala=tada into the environment and checking for that would be an idea; but since I restore the environment right away, there is no way I can see to actually access the new os.environ - [x] Passes isort -rc . && black . && mypy . && flake8 - [x] Fully documented, including whats-new.rst for all changes and api.rst for new API

I added a section to the relevant docstring. Not sure how much this needs to also be included in the other files.

Hello,

I'd like to open some old-school Grib files from one of our climate models. I'm using the PyNIO backend for this; which works pretty well so far -- at least the data opens! However, I am getting errors:

warning:NclGRIB: Unrecognized parameter table (center 252, subcenter 1, table 128), defaulting to NCEP operational table: variable names and units may be incorrect

So, would it somehow be possible to provide a code table to be used when opening grb files? I have files next to my output where the codes are stored. An example is below.

I can imagine something like:

python ds = xr.open_dataset("/path/to/file.grb", engine="pynio", code_table="/path/to/codetab")

Would this be difficult to implement? Cheers, Paul

130 47 st 0.00 1.00 temperature [K] 138 47 svo 0.00 1.00 vorticity [1/s] 152 1 lsp 0.00 1.00 log surface pressure [] 155 47 sd 0.00 1.00 divergence [1/s] 133 47 q 0.00 1.00 specific humidity [kg/kg] 153 47 xl 0.00 1.00 cloud water [kg/kg] 154 47 xi 0.00 1.00 cloud ice [kg/kg] 50 1 rsdscs 0.00 1.00 surface downwelling shortwave radiation flux (clear-sky) [W/m**2] 51 1 rsuscs 0.00 1.00 surface upwelling shortwave radiation flux (clear-sky) [W/m**2] 52 1 rsdscs_na 0.00 1.00 instantaneous surface downwelling shortwave radiation flux (clear-sky) [W/m**2] 53 1 rsuscs_na 0.00 1.00 instantaneous surface upwelling shortwave radiation flux (clear-sky) [W/m**2] 54 1 q2m 0.00 1.00 2m specific humidity [] 55 1 rh2m 0.00 1.00 2m relative humidity [] 56 1 rsdsiac 0.00 1.00 surface downwelling shortwave radiation over ice [W/m**2] 57 1 rsdswac 0.00 1.00 surface downwelling shortwave radiation over water [W/m**2] 58 1 rsusiac 0.00 1.00 surface upwelling shortwave radiation over ice [W/m**2] 59 1 rsuswac 0.00 1.00 surface upwelling shortwave radiation over water [W/m**2] 60 1 rldsiac 0.00 1.00 surface downwelling longwave radiation over ice [W/m**2] 61 1 rldswac 0.00 1.00 surface downwelling longwave radiation over water [W/m**2] 62 1 rlusiac 0.00 1.00 surface upwelling longwave radiation over ice [W/m**2] 63 1 rluswac 0.00 1.00 surface upwelling longwave radiation over water [W/m**2] 64 1 sh_vdiff 0.00 1.00 column heating due to vertical diffusion [W/m**2] 65 1 ev_vdiff 0.00 1.00 column moistening due to vertical diffusion [kg/m**2s] 66 1 ch_concloud 0.00 1.00 convective heating [W/m**s] 67 1 cw_concloud 0.00 1.00 convective moistening [kg/m**2s] 68 1 fage 0.00 1.00 aging factor of snow on ice [] 69 1 snifrac 0.00 1.00 fraction of ice covered with snow [] 70 1 barefrac 0.00 1.00 bare ice fraction [] 71 1 alsom 0.00 1.00 albedo of melt ponds [] 72 1 alsobs 0.00 1.00 albedo of bare ice and snow without ponds [] 73 1 sicepdw 0.00 1.00 melt pond depth on sea-ice [m] 74 1 sicepdi 0.00 1.00 ice thickness on melt pond [m] 75 1 tsicepdi 0.00 1.00 ice temperature on frozen melt pond [K] 76 1 sicepres 0.00 1.00 residual heat flux [W/m**2] 77 1 ameltdepth 0.00 1.00 total melt pond depth [m] 78 1 ameltfrac 0.00 1.00 fract area of melt ponds on sea-ice [] 79 1 albedo_vis_dir 0.00 1.00 surface albedo visible range direct [] 80 1 albedo_nir_dir 0.00 1.00 surface albedo NIR range direct [] 81 1 albedo_vis_dif 0.00 1.00 surface albedo visible range diffuse [] 82 1 albedo_nir_dif 0.00 1.00 surface albedo NIR range diffuse [] 83 1 ocu 0.00 1.00 ocean eastward velocity [m/s] 84 1 ocv 0.00 1.00 ocean northward velocity [m/s] 85 1 tradl 0.00 1.00 thermal radiation 200mb [W/m**2] 86 1 sradl 0.00 1.00 solar radiation 200mb [W/m**2] 87 1 trafl 0.00 1.00 thermal radiation 200mb (clear sky) [W/m**2] 88 1 srafl 0.00 1.00 solar radiation 200mb (clear sky) [W/m**2] 89 1 amlcorac 0.00 1.00 mixed layer flux correction [W/m**2] 91 1 trfliac 0.00 1.00 LW flux over ice [W/m**2] 92 1 trflwac 0.00 1.00 LW flux over water [W/m**2] 93 1 trfllac 0.00 1.00 LW flux over land [W/m**2] 94 1 sofliac 0.00 1.00 SW flux over ice [W/m**2] 95 1 soflwac 0.00 1.00 SW flux over water [W/m**2] 96 1 sofllac 0.00 1.00 SW flux over land [W/m**2] 97 1 friac 0.00 1.00 ice cover (fraction of grid box) [] 100 1 albedo_vis 0.00 1.00 surface albedo visible range [] 101 1 albedo_nir 0.00 1.00 surface albedo NIR range [] 102 1 tsi 0.00 1.00 surface temperature of ice [K] 103 1 tsw 0.00 1.00 surface temperature of water [K] 104 1 ustri 0.00 1.00 zonal wind stress over ice [Pa] 105 1 vstri 0.00 1.00 meridional wind stress over ice [Pa] 106 1 ustrw 0.00 1.00 zonal wind stress over water [Pa] 107 1 vstrw 0.00 1.00 meridional wind stress over water [Pa] 108 1 ustrl 0.00 1.00 zonal wind stress over land [Pa] 109 1 vstrl 0.00 1.00 meridional wind stress over land [Pa] 110 1 ahfliac 0.00 1.00 latent heat flux over ice [W/m**2] 111 1 ahflwac 0.00 1.00 latent heat flux over water [W/m**2] 112 1 ahfllac 0.00 1.00 latent heat flux over land [W/m**2] 113 1 evapiac 0.00 1.00 evaporation over ice [kg/m**2s] 114 1 evapwac 0.00 1.00 evaporation over water [kg/m**2s] 115 1 evaplac 0.00 1.00 evaporation over land [kg/m**2s] 116 1 az0i 0.00 1.00 roughness length over ice [m] 117 1 az0w 0.00 1.00 roughness length over water [m] 118 1 az0l 0.00 1.00 roughness length over land [m] 119 1 ahfsiac 0.00 1.00 sensible heat flux over ice [W/m**2] 120 1 ahfswac 0.00 1.00 sensible heat flux over water [W/m**2] 121 1 ahfslac 0.00 1.00 sensible heat flux over land [W/m**2] 122 1 alsoi 0.00 1.00 albedo of ice [] 123 1 alsow 0.00 1.00 albedo of water [] 124 1 alsol 0.00 1.00 albedo of land [] 125 1 ahfice 0.00 1.00 conductive heat flux [W/m**2] 126 1 qres 0.00 1.00 residual heat flux for melting sea ice [W/m**2] 129 1 geosp 0.00 1.00 surface geopotential (orography) [m**2/s**2] 134 1 aps 0.00 1.00 surface pressure [Pa] 137 1 apmeb 0.00 1.00 vertic integr tendenc of water [kg/m**2s] 139 1 tslm1 0.00 1.00 surface temperature of land [K] 140 1 ws 0.00 1.00 soil wetness [m] 141 1 sn 0.00 1.00 snow depth [m] 142 1 aprl 0.00 1.00 large scale precipitation [kg/m**2s] 143 1 aprc 0.00 1.00 convective precipitation [kg/m**2s] 144 1 aprs 0.00 1.00 snow fall [kg/m**2s] 145 1 vdis 0.00 1.00 boundary layer dissipation [W/m**2] 146 1 ahfs 0.00 1.00 sensible heat flux [W/m**2] 147 1 ahfl 0.00 1.00 latent heat flux [W/m**2] 150 1 xivi 0.00 1.00 vertically integrated cloud ice [kg/m**2] 157 47 relhum 0.00 1.00 relative humidity [] 164 1 aclcov 0.00 1.00 total cloud cover [] 165 1 u10 0.00 1.00 10m u-velocity [m/s] 166 1 v10 0.00 1.00 10m v-velocity [m/s] 167 1 temp2 0.00 1.00 2m temperature [K] 168 1 dew2 0.00 1.00 2m dew point temperature [K] 169 1 tsurf 0.00 1.00 surface temperature [K] 171 1 wind10 0.00 1.00 10m windspeed [m/s] 172 1 slm 0.00 1.00 land sea mask (1=land, 0=sea/lakes) [] 175 1 albedo 0.00 1.00 surface albedo [] 176 1 srads 0.00 1.00 net surface solar radiation [W/m**2] 177 1 trads 0.00 1.00 net surface thermal radiation [W/m**2] 178 1 srad0 0.00 1.00 net top solar radiation [W/m**2] 179 1 trad0 0.00 1.00 top thermal radiation (OLR) [W/m**2] 180 1 ustr 0.00 1.00 u-stress [Pa] 181 1 vstr 0.00 1.00 v-stress [Pa] 182 1 evap 0.00 1.00 evaporation [kg/m**2s] 184 1 srad0d 0.00 1.00 top incoming solar radiation [W/m**2] 185 1 srafs 0.00 1.00 net surface solar radiation (clear sky) [W/m**2] 186 1 trafs 0.00 1.00 net surface therm radiation (clear sky) [W/m**2] 187 1 sraf0 0.00 1.00 net top solar radiation (clear sky) [W/m**2] 188 1 traf0 0.00 1.00 net top thermal radiation (clear sky) [W/m**2] 193 1 wl 0.00 1.00 skin reservoir content [m] 194 1 slf 0.00 1.00 sea land fraction [] 197 1 vdisgw 0.00 1.00 gravity wave dissipation [W/m**2] 201 1 t2max 0.00 1.00 maximum 2m temperature [K] 202 1 t2min 0.00 1.00 minimum 2m temperature [K] 203 1 srad0u 0.00 1.00 top solar radiation upward [W/m**2] 204 1 sradsu 0.00 1.00 surface solar radiation upward [W/m**2] 205 1 tradsu 0.00 1.00 surface thermal radiation upward [W/m**2] 208 1 ahfcon 0.00 1.00 conductive heat flux through ice [W/m**2] 209 1 ahfres 0.00 1.00 melting of ice [W/m**2] 210 1 seaice 0.00 1.00 ice cover (fraction of 1-SLM) [] 211 1 siced 0.00 1.00 ice depth [m] 213 1 gld 0.00 1.00 glacier depth [m] 214 1 sni 0.00 1.00 water equivalent of snow on ice [m] 216 1 wimax 0.00 1.00 maximum 10m-wind speed [m/s] 217 1 topmax 0.00 1.00 max height of conv cloud tops [Pa] 223 47 aclcac 0.00 1.00 cloud cover [] 229 1 wsmx 0.00 1.00 field capacity of soil [m] 230 1 qvi 0.00 1.00 vertically integrated water vapor [kg/m**2] 231 1 xlvi 0.00 1.00 vertically integrated cloud water [kg/m**2] 232 1 glac 0.00 1.00 fraction of land covered by glaciers [] 233 1 snc 0.00 1.00 snow depth at the canopy [m] 235 1 abso4 0.00 1.00 antropogenic sulfur burden [kg/m**2] 236 47 ao3 0.00 1.00 ipcc ozone [kg/kg] 237 1 tropo 0.00 1.00 WMO defined tropopause height [Pa] 238 1 thvsig 0.00 1.00 stddev virt pot temp at halflev klevm1 [K] 239 47 tpot 0.00 1.00 potential temperature [K]

I recently learned that tqdm can automatically create a progress bar for you when you are doing expensive groupby/map operations. See for instance here: https://stackoverflow.com/questions/18603270/progress-indicator-during-pandas-operations

Would it be simple to implement something similar in Xarray? The documentation seems to read as if the groupby is heavily inspired by pandas.

Is your feature request related to a problem? Please describe. No, everything works as expected, this would just be a "quality of life" improvement.

Describe the solution you'd like

The implementation in tqdm states: Registers the given `tqdm` instance with `pandas.core.groupby.DataFrameGroupBy.progress_apply`.

I suppose something similar would need to be implemented in Xarray, and then we might be able to copy the tqdm logic.

Describe alternatives you've considered I could loop over whatever dimension I have and make my own progress bar, but that seems like defeating the purpose of groupby.

Code Sample, a copy-pastable example if possible

It appears as if the open_mfdataset cannot handle many files (many here = 1200)

```python ensemble = xr.open_mfdataset("/scratch/simulation_database/incoming/Eem125-S2/output/Eem125-S2_echam5_main_mm_26*.nc")

OSError Traceback (most recent call last) <ipython-input-4-038705c4f255> in <module>() ----> 1 ensemble = xr.open_mfdataset("/scratch/simulation_database/incoming/Eem125-S2/output/Eem125-S2_echam5_main_mm_26*.nc")

~/anaconda3/lib/python3.6/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, **kwargs)

~/anaconda3/lib/python3.6/site-packages/xarray/backends/api.py in <listcomp>(.0)

~/anaconda3/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables)

~/anaconda3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py in open(cls, filename, mode, format, group, writer, clobber, diskless, persist, autoclose)

~/anaconda3/lib/python3.6/site-packages/xarray/backends/netCDF4_.py in _open_netcdf4_group(filename, mode, group, **kwargs)

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.init()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

OSError: [Errno 24] Too many open files: b'/scratch/simulation_database/incoming/Eem125-S2/output/Eem125-S2_echam5_main_mm_260001.nc' ```

Problem description

Often, climate simulations produce more than one output file per model component (generally 1 per saved time output, e.g. months, years, days, or something else). It would be good to access all of these as one object, rather than having to combining them by hand before with e.g. cdo or some other tool.

Expected Output

ensemble variable definition gives me back 1 object 😄

Output of xr.show_versions()