html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1704#issuecomment-346028156,https://api.github.com/repos/pydata/xarray/issues/1704,346028156,MDEyOklzc3VlQ29tbWVudDM0NjAyODE1Ng==,14314623,2017-11-21T13:35:38Z,2017-11-21T13:35:38Z,CONTRIBUTOR,"Yes that would be phenomenal. It would be absolutely sufficient for this to be an optional step that needs to be activated with e.g. xr.open_mfdataset(filelist, full_check=True).
In my particular use case I encountered very long load times, and it would be great if in any such problematic cases there would be an easy way to activate a suite of verbose checks to quickly identify problems. For normal use these could be largely deactivated to achieve a fast and streamlined readin?
Sent from my iPhone
> On Nov 21, 2017, at 12:17 AM, Joe Hamman wrote:
>
> @jbusecke -
>
> ... would it be possible to implement a check for such errors that displays a warning?
>
> In your case, what would that have looked like? Many users have actually requested that xarray provide a more streamlined reader function that does less checking. Perhaps it would have been useful for you to have xarray raise an error when it encountered files that weren't consistent (either in the variable, dimensions, or coordinates)?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,272415954
https://github.com/pydata/xarray/issues/1704#issuecomment-344352515,https://api.github.com/repos/pydata/xarray/issues/1704,344352515,MDEyOklzc3VlQ29tbWVudDM0NDM1MjUxNQ==,14314623,2017-11-14T18:28:51Z,2017-11-14T18:28:51Z,CONTRIBUTOR,"I believe I solved the issue: It turns out the reason for the slow performance were inconsistencies in between different files (added data_vars and coords). Specifying `drop_variables` yields the expected performance.
I am not sure if that is a feasible option, but would it be possible to implement a check for such errors that displays a warning?
The error regarding `enige='scipy'` remains but I believe this is due to changing netcdf formats between files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,272415954
https://github.com/pydata/xarray/issues/1704#issuecomment-343344036,https://api.github.com/repos/pydata/xarray/issues/1704,343344036,MDEyOklzc3VlQ29tbWVudDM0MzM0NDAzNg==,14314623,2017-11-10T01:11:00Z,2017-11-10T01:11:00Z,CONTRIBUTOR,"Ok I just tried to read the files with the new RC. Same error.
here is the output of `xr.show_versions()`:
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-642.15.1.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US
LOCALE: en_US.ISO8859-1
xarray: 0.10.0rc1
pandas: 0.21.0
numpy: 1.13.3
scipy: 0.19.1
netCDF4: 1.3.1
h5netcdf: None
Nio: None
bottleneck: None
cyordereddict: None
dask: 0.15.4
matplotlib: 2.1.0
cartopy: 0.15.1
seaborn: 0.8.1
setuptools: 36.6.0
pip: 9.0.1
conda: None
pytest: 3.2.3
IPython: 6.2.1
sphinx: None
```
and here is an `ncdump -h` of one of the files:
```
$ ncdump -h ocean.0198.ann.nc
netcdf ocean.0198.ann {
dimensions:
xt_ocean = 3600 ;
yt_ocean = 2700 ;
time = UNLIMITED ; // (1 currently)
nv = 2 ;
xu_ocean = 3600 ;
yu_ocean = 2700 ;
st_ocean = 50 ;
st_edges_ocean = 51 ;
sw_edges_ocean = 51 ;
sw_ocean = 50 ;
variables:
double xt_ocean(xt_ocean) ;
xt_ocean:long_name = ""tcell longitude"" ;
xt_ocean:units = ""degrees_E"" ;
xt_ocean:cartesian_axis = ""X"" ;
double yt_ocean(yt_ocean) ;
yt_ocean:long_name = ""tcell latitude"" ;
yt_ocean:units = ""degrees_N"" ;
yt_ocean:cartesian_axis = ""Y"" ;
double time(time) ;
time:long_name = ""time"" ;
time:units = ""days since 0001-01-01 00:00:00"" ;
time:cartesian_axis = ""T"" ;
time:calendar_type = ""JULIAN"" ;
time:calendar = ""JULIAN"" ;
time:bounds = ""time_bounds"" ;
double nv(nv) ;
nv:long_name = ""vertex number"" ;
nv:units = ""none"" ;
nv:cartesian_axis = ""N"" ;
double xu_ocean(xu_ocean) ;
xu_ocean:long_name = ""ucell longitude"" ;
xu_ocean:units = ""degrees_E"" ;
xu_ocean:cartesian_axis = ""X"" ;
double yu_ocean(yu_ocean) ;
yu_ocean:long_name = ""ucell latitude"" ;
yu_ocean:units = ""degrees_N"" ;
yu_ocean:cartesian_axis = ""Y"" ;
double st_ocean(st_ocean) ;
st_ocean:long_name = ""tcell zstar depth"" ;
st_ocean:units = ""meters"" ;
st_ocean:cartesian_axis = ""Z"" ;
st_ocean:positive = ""down"" ;
st_ocean:edges = ""st_edges_ocean"" ;
double st_edges_ocean(st_edges_ocean) ;
st_edges_ocean:long_name = ""tcell zstar depth edges"" ;
st_edges_ocean:units = ""meters"" ;
st_edges_ocean:cartesian_axis = ""Z"" ;
st_edges_ocean:positive = ""down"" ;
double sw_edges_ocean(sw_edges_ocean) ;
sw_edges_ocean:long_name = ""ucell zstar depth edges"" ;
sw_edges_ocean:units = ""meters"" ;
sw_edges_ocean:cartesian_axis = ""Z"" ;
sw_edges_ocean:positive = ""down"" ;
float geolon_t(yt_ocean, xt_ocean) ;
geolon_t:long_name = ""tracer longitude"" ;
geolon_t:units = ""degrees_E"" ;
geolon_t:valid_range = -281.f, 361.f ;
geolon_t:missing_value = 1.e+20f ;
geolon_t:_FillValue = 1.e+20f ;
geolon_t:cell_methods = ""time: point"" ;
geolon_t:coordinates = ""geolon_t geolat_t"" ;
float geolat_t(yt_ocean, xt_ocean) ;
geolat_t:long_name = ""tracer latitude"" ;
geolat_t:units = ""degrees_N"" ;
geolat_t:valid_range = -91.f, 91.f ;
geolat_t:missing_value = 1.e+20f ;
geolat_t:_FillValue = 1.e+20f ;
geolat_t:cell_methods = ""time: point"" ;
geolat_t:coordinates = ""geolon_t geolat_t"" ;
float geolon_c(yu_ocean, xu_ocean) ;
geolon_c:long_name = ""uv longitude"" ;
geolon_c:units = ""degrees_E"" ;
geolon_c:valid_range = -281.f, 361.f ;
geolon_c:missing_value = 1.e+20f ;
geolon_c:_FillValue = 1.e+20f ;
geolon_c:cell_methods = ""time: point"" ;
geolon_c:coordinates = ""geolon_c geolat_c"" ;
float geolat_c(yu_ocean, xu_ocean) ;
geolat_c:long_name = ""uv latitude"" ;
geolat_c:units = ""degrees_N"" ;
geolat_c:valid_range = -91.f, 91.f ;
geolat_c:missing_value = 1.e+20f ;
geolat_c:_FillValue = 1.e+20f ;
geolat_c:cell_methods = ""time: point"" ;
geolat_c:coordinates = ""geolon_c geolat_c"" ;
float temp(time, st_ocean, yt_ocean, xt_ocean) ;
temp:long_name = ""Potential temperature"" ;
temp:units = ""degrees C"" ;
temp:valid_range = -10.f, 500.f ;
temp:missing_value = -1.e+20f ;
temp:_FillValue = -1.e+20f ;
temp:cell_methods = ""time: mean"" ;
temp:time_avg_info = ""average_T1,average_T2,average_DT"" ;
temp:coordinates = ""geolon_t geolat_t"" ;
temp:standard_name = ""sea_water_potential_temperature"" ;
double time_bounds(time, nv) ;
time_bounds:long_name = ""time axis boundaries"" ;
time_bounds:units = ""days"" ;
time_bounds:missing_value = 1.e+20 ;
time_bounds:_FillValue = 1.e+20 ;
float salt(time, st_ocean, yt_ocean, xt_ocean) ;
salt:long_name = ""Practical Salinity"" ;
salt:units = ""psu"" ;
salt:valid_range = -10.f, 100.f ;
salt:missing_value = -1.e+20f ;
salt:_FillValue = -1.e+20f ;
salt:cell_methods = ""time: mean"" ;
salt:time_avg_info = ""average_T1,average_T2,average_DT"" ;
salt:coordinates = ""geolon_t geolat_t"" ;
salt:standard_name = ""sea_water_salinity"" ;
float u(time, st_ocean, yu_ocean, xu_ocean) ;
u:long_name = ""i-current"" ;
u:units = ""m/sec"" ;
u:valid_range = -10.f, 10.f ;
u:missing_value = -1.e+20f ;
u:_FillValue = -1.e+20f ;
u:cell_methods = ""time: mean"" ;
u:time_avg_info = ""average_T1,average_T2,average_DT"" ;
u:coordinates = ""geolon_c geolat_c"" ;
u:standard_name = ""sea_water_x_velocity"" ;
float v(time, st_ocean, yu_ocean, xu_ocean) ;
v:long_name = ""j-current"" ;
v:units = ""m/sec"" ;
v:valid_range = -10.f, 10.f ;
v:missing_value = -1.e+20f ;
v:_FillValue = -1.e+20f ;
v:cell_methods = ""time: mean"" ;
v:time_avg_info = ""average_T1,average_T2,average_DT"" ;
v:coordinates = ""geolon_c geolat_c"" ;
v:standard_name = ""sea_water_y_velocity"" ;
float pot_rho_0(time, st_ocean, yt_ocean, xt_ocean) ;
pot_rho_0:long_name = ""potential density referenced to 0 dbar"" ;
pot_rho_0:units = ""kg/m^3"" ;
pot_rho_0:valid_range = -10.f, 100000.f ;
pot_rho_0:missing_value = -1.e+20f ;
pot_rho_0:_FillValue = -1.e+20f ;
pot_rho_0:cell_methods = ""time: mean"" ;
pot_rho_0:time_avg_info = ""average_T1,average_T2,average_DT"" ;
pot_rho_0:coordinates = ""geolon_t geolat_t"" ;
pot_rho_0:standard_name = ""sea_water_potential_density"" ;
float ty_trans(time, st_ocean, yu_ocean, xt_ocean) ;
ty_trans:long_name = ""T-cell j-mass transport"" ;
ty_trans:units = ""Sv (10^9 kg/s)"" ;
ty_trans:valid_range = -1.e+20f, 1.e+20f ;
ty_trans:missing_value = -1.e+20f ;
ty_trans:_FillValue = -1.e+20f ;
ty_trans:cell_methods = ""time: mean"" ;
ty_trans:time_avg_info = ""average_T1,average_T2,average_DT"" ;
ty_trans:coordinates = ""geolon_t geolat_c"" ;
ty_trans:standard_name = ""ocean_y_mass_transport"" ;
float eta_t(time, yt_ocean, xt_ocean) ;
eta_t:long_name = ""surface height on T cells [Boussinesq (volume conserving) model]"" ;
eta_t:units = ""meter"" ;
eta_t:valid_range = -1000.f, 1000.f ;
eta_t:missing_value = -1.e+20f ;
eta_t:_FillValue = -1.e+20f ;
eta_t:cell_methods = ""time: mean"" ;
eta_t:time_avg_info = ""average_T1,average_T2,average_DT"" ;
eta_t:coordinates = ""geolon_t geolat_t"" ;
float eta_u(time, yu_ocean, xu_ocean) ;
eta_u:long_name = ""surface height on U cells"" ;
eta_u:units = ""meter"" ;
eta_u:valid_range = -1000.f, 1000.f ;
eta_u:missing_value = -1.e+20f ;
eta_u:_FillValue = -1.e+20f ;
eta_u:cell_methods = ""time: mean"" ;
eta_u:time_avg_info = ""average_T1,average_T2,average_DT"" ;
eta_u:coordinates = ""geolon_c geolat_c"" ;
float frazil_2d(time, yt_ocean, xt_ocean) ;
frazil_2d:long_name = ""ocn frazil heat flux over time step"" ;
frazil_2d:units = ""W/m^2"" ;
frazil_2d:valid_range = -1.e+10f, 1.e+10f ;
frazil_2d:missing_value = -1.e+20f ;
frazil_2d:_FillValue = -1.e+20f ;
frazil_2d:cell_methods = ""time: mean"" ;
frazil_2d:time_avg_info = ""average_T1,average_T2,average_DT"" ;
frazil_2d:coordinates = ""geolon_t geolat_t"" ;
float hblt(time, yt_ocean, xt_ocean) ;
hblt:long_name = ""T-cell boundary layer depth from KPP"" ;
hblt:units = ""m"" ;
hblt:valid_range = -100000.f, 1000000.f ;
hblt:missing_value = -1.e+20f ;
hblt:_FillValue = -1.e+20f ;
hblt:cell_methods = ""time: mean"" ;
hblt:time_avg_info = ""average_T1,average_T2,average_DT"" ;
hblt:coordinates = ""geolon_t geolat_t"" ;
hblt:standard_name = ""ocean_mixed_layer_thickness_defined_by_mixing_scheme"" ;
float mld(time, yt_ocean, xt_ocean) ;
mld:long_name = ""mixed layer depth determined by density criteria"" ;
mld:units = ""m"" ;
mld:valid_range = 0.f, 1000000.f ;
mld:missing_value = -1.e+20f ;
mld:_FillValue = -1.e+20f ;
mld:cell_methods = ""time: mean"" ;
mld:time_avg_info = ""average_T1,average_T2,average_DT"" ;
mld:coordinates = ""geolon_t geolat_t"" ;
mld:standard_name = ""ocean_mixed_layer_thickness_defined_by_sigma_t"" ;
float mld_dtheta(time, yt_ocean, xt_ocean) ;
mld_dtheta:long_name = ""mixed layer depth determined by temperature criteria"" ;
mld_dtheta:units = ""m"" ;
mld_dtheta:valid_range = 0.f, 1000000.f ;
mld_dtheta:missing_value = -1.e+20f ;
mld_dtheta:_FillValue = -1.e+20f ;
mld_dtheta:cell_methods = ""time: mean"" ;
mld_dtheta:time_avg_info = ""average_T1,average_T2,average_DT"" ;
mld_dtheta:coordinates = ""geolon_t geolat_t"" ;
float net_sfc_heating(time, yt_ocean, xt_ocean) ;
net_sfc_heating:long_name = ""surface ocean heat flux coming through coupler and mass transfer"" ;
net_sfc_heating:units = ""Watts/m^2"" ;
net_sfc_heating:valid_range = -10000.f, 10000.f ;
net_sfc_heating:missing_value = -1.e+20f ;
net_sfc_heating:_FillValue = -1.e+20f ;
net_sfc_heating:cell_methods = ""time: mean"" ;
net_sfc_heating:time_avg_info = ""average_T1,average_T2,average_DT"" ;
net_sfc_heating:coordinates = ""geolon_t geolat_t"" ;
float pme_river(time, yt_ocean, xt_ocean) ;
pme_river:long_name = ""mass flux of precip-evap+river via sbc (liquid, frozen, evaporation)"" ;
pme_river:units = ""(kg/m^3)*(m/sec)"" ;
pme_river:valid_range = -1000000.f, 1000000.f ;
pme_river:missing_value = -1.e+20f ;
pme_river:_FillValue = -1.e+20f ;
pme_river:cell_methods = ""time: mean"" ;
pme_river:time_avg_info = ""average_T1,average_T2,average_DT"" ;
pme_river:coordinates = ""geolon_t geolat_t"" ;
pme_river:standard_name = ""water_flux_into_sea_water"" ;
float river(time, yt_ocean, xt_ocean) ;
river:long_name = ""mass flux of river (runoff + calving) entering ocean"" ;
river:units = ""(kg/m^3)*(m/sec)"" ;
river:valid_range = -1000000.f, 1000000.f ;
river:missing_value = -1.e+20f ;
river:_FillValue = -1.e+20f ;
river:cell_methods = ""time: mean"" ;
river:time_avg_info = ""average_T1,average_T2,average_DT"" ;
river:coordinates = ""geolon_t geolat_t"" ;
float salt_int_rhodz(time, yt_ocean, xt_ocean) ;
salt_int_rhodz:long_name = ""vertical sum of Practical Salinity * rho_dzt"" ;
salt_int_rhodz:units = ""psu*(kg/m^3)*m"" ;
salt_int_rhodz:valid_range = -1.e+20f, 1.e+20f ;
salt_int_rhodz:missing_value = -1.e+20f ;
salt_int_rhodz:_FillValue = -1.e+20f ;
salt_int_rhodz:cell_methods = ""time: mean"" ;
salt_int_rhodz:time_avg_info = ""average_T1,average_T2,average_DT"" ;
salt_int_rhodz:coordinates = ""geolon_t geolat_t"" ;
float sea_level(time, yt_ocean, xt_ocean) ;
sea_level:long_name = ""effective sea level (eta_t + patm/(rho0*g)) on T cells"" ;
sea_level:units = ""meter"" ;
sea_level:valid_range = -1000.f, 1000.f ;
sea_level:missing_value = -1.e+20f ;
sea_level:_FillValue = -1.e+20f ;
sea_level:cell_methods = ""time: mean"" ;
sea_level:time_avg_info = ""average_T1,average_T2,average_DT"" ;
sea_level:coordinates = ""geolon_t geolat_t"" ;
sea_level:standard_name = ""sea_surface_height_above_geoid"" ;
float sea_levelsq(time, yt_ocean, xt_ocean) ;
sea_levelsq:long_name = ""square of effective sea level (eta_t + patm/(rho0*g)) on T cells"" ;
sea_levelsq:units = ""m^2"" ;
sea_levelsq:valid_range = -1000.f, 1000.f ;
sea_levelsq:missing_value = -1.e+20f ;
sea_levelsq:_FillValue = -1.e+20f ;
sea_levelsq:cell_methods = ""time: mean"" ;
sea_levelsq:time_avg_info = ""average_T1,average_T2,average_DT"" ;
sea_levelsq:coordinates = ""geolon_t geolat_t"" ;
sea_levelsq:standard_name = ""square_of_sea_surface_height_above_geoid"" ;
float sfc_hflux_coupler(time, yt_ocean, xt_ocean) ;
sfc_hflux_coupler:long_name = ""surface heat flux coming through coupler"" ;
sfc_hflux_coupler:units = ""Watts/m^2"" ;
sfc_hflux_coupler:valid_range = -10000.f, 10000.f ;
sfc_hflux_coupler:missing_value = -1.e+20f ;
sfc_hflux_coupler:_FillValue = -1.e+20f ;
sfc_hflux_coupler:cell_methods = ""time: mean"" ;
sfc_hflux_coupler:time_avg_info = ""average_T1,average_T2,average_DT"" ;
sfc_hflux_coupler:coordinates = ""geolon_t geolat_t"" ;
double sw_ocean(sw_ocean) ;
sw_ocean:long_name = ""ucell zstar depth"" ;
sw_ocean:units = ""meters"" ;
sw_ocean:cartesian_axis = ""Z"" ;
sw_ocean:positive = ""down"" ;
sw_ocean:edges = ""sw_edges_ocean"" ;
float tau_x(time, yu_ocean, xu_ocean) ;
tau_x:long_name = ""i-directed wind stress forcing u-velocity"" ;
tau_x:units = ""N/m^2"" ;
tau_x:valid_range = -10.f, 10.f ;
tau_x:missing_value = -1.e+20f ;
tau_x:_FillValue = -1.e+20f ;
tau_x:cell_methods = ""time: mean"" ;
tau_x:time_avg_info = ""average_T1,average_T2,average_DT"" ;
tau_x:coordinates = ""geolon_c geolat_c"" ;
tau_x:standard_name = ""surface_downward_x_stress"" ;
float tau_y(time, yu_ocean, xu_ocean) ;
tau_y:long_name = ""j-directed wind stress forcing v-velocity"" ;
tau_y:units = ""N/m^2"" ;
tau_y:valid_range = -10.f, 10.f ;
tau_y:missing_value = -1.e+20f ;
tau_y:_FillValue = -1.e+20f ;
tau_y:cell_methods = ""time: mean"" ;
tau_y:time_avg_info = ""average_T1,average_T2,average_DT"" ;
tau_y:coordinates = ""geolon_c geolat_c"" ;
tau_y:standard_name = ""surface_downward_y_stress"" ;
float temp_int_rhodz(time, yt_ocean, xt_ocean) ;
temp_int_rhodz:long_name = ""vertical sum of Potential temperature * rho_dzt"" ;
temp_int_rhodz:units = ""deg_C*(kg/m^3)*m"" ;
temp_int_rhodz:valid_range = -1.e+20f, 1.e+20f ;
temp_int_rhodz:missing_value = -1.e+20f ;
temp_int_rhodz:_FillValue = -1.e+20f ;
temp_int_rhodz:cell_methods = ""time: mean"" ;
temp_int_rhodz:time_avg_info = ""average_T1,average_T2,average_DT"" ;
temp_int_rhodz:coordinates = ""geolon_t geolat_t"" ;
float wt(time, sw_ocean, yt_ocean, xt_ocean) ;
wt:long_name = ""dia-surface velocity T-points"" ;
wt:units = ""m/sec"" ;
wt:valid_range = -100000.f, 100000.f ;
wt:missing_value = -1.e+20f ;
wt:_FillValue = -1.e+20f ;
wt:cell_methods = ""time: mean"" ;
wt:time_avg_info = ""average_T1,average_T2,average_DT"" ;
wt:coordinates = ""geolon_t geolat_t"" ;
// global attributes:
:filename = ""01980101.ocean.nc"" ;
:title = ""CM2.6_miniBling"" ;
:grid_type = ""mosaic"" ;
:grid_tile = ""1"" ;
:history = ""Tue Feb 25 16:32:17 2014: ncks --64bit --hdr_pad 15000 -A ocean.0198.ann2d.nc ocean.0198.ann.nc\n"",
""Tue Feb 25 16:23:35 2014: ncks --64bit --hdr_pad 15000 -A frazil_2d.nc ocean.0198.ann2d.nc\n"",
""Tue Feb 25 16:23:32 2014: ncra -O -v nv,time_bounds,geolat_c,geolat_t,geolon_c,geolon_t,st_edges_ocean,sw_edges_ocean,frazil_2d 01980101.ocean.nc frazil_2d.nc"" ;
:nco_openmp_thread_number = 1 ;
:NCO = ""4.1.0"" ;
}
```
I am unsure if I am allowed to share them publicly, is there another way to diagnose what is going on?
Thanks a lot!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,272415954