issue_comments

Closing in favour of #1823

Yes that would be phenomenal. It would be absolutely sufficient for this to be an optional step that needs to be activated with e.g. xr.open_mfdataset(filelist, full_check=True). In my particular use case I encountered very long load times, and it would be great if in any such problematic cases there would be an easy way to activate a suite of verbose checks to quickly identify problems. For normal use these could be largely deactivated to achieve a fast and streamlined readin?

Sent from my iPhone

On Nov 21, 2017, at 12:17 AM, Joe Hamman notifications@github.com wrote:

@jbusecke -

... would it be possible to implement a check for such errors that displays a warning?

In your case, what would that have looked like? Many users have actually requested that xarray provide a more streamlined reader function that does less checking. Perhaps it would have been useful for you to have xarray raise an error when it encountered files that weren't consistent (either in the variable, dimensions, or coordinates)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

@jbusecke -

... would it be possible to implement a check for such errors that displays a warning?

In your case, what would that have looked like? Many users have actually requested that xarray provide a more streamlined reader function that does less checking. Perhaps it would have been useful for you to have xarray raise an error when it encountered files that weren't consistent (either in the variable, dimensions, or coordinates)?

I believe I solved the issue: It turns out the reason for the slow performance were inconsistencies in between different files (added data_vars and coords). Specifying drop_variables yields the expected performance.

I am not sure if that is a feasible option, but would it be possible to implement a check for such errors that displays a warning?

The error regarding enige='scipy' remains but I believe this is due to changing netcdf formats between files.

Can you figure out which variable name it's erroring on? Try dropping into a debugger (e.g., %debug% in IPython).

It is not immediately obvious to me what the issue is with this file, but I suspect this is probably a SciPy issue given that scipy.io.netcdf_file() cannot even open it. Access to the file would help because then I can use a debugger to see exactly what went wrong.

Ok I just tried to read the files with the new RC. Same error.

here is the output of xr.show_versions(): ``` INSTALLED VERSIONS

commit: None python: 3.6.3.final.0 python-bits: 64 OS: Linux OS-release: 2.6.32-642.15.1.el6.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US LOCALE: en_US.ISO8859-1

xarray: 0.10.0rc1 pandas: 0.21.0 numpy: 1.13.3 scipy: 0.19.1 netCDF4: 1.3.1 h5netcdf: None Nio: None bottleneck: None cyordereddict: None dask: 0.15.4 matplotlib: 2.1.0 cartopy: 0.15.1 seaborn: 0.8.1 setuptools: 36.6.0 pip: 9.0.1 conda: None pytest: 3.2.3 IPython: 6.2.1 sphinx: None ```

and here is an ncdump -h of one of the files: ``` $ ncdump -h ocean.0198.ann.nc netcdf ocean.0198.ann { dimensions: xt_ocean = 3600 ; yt_ocean = 2700 ; time = UNLIMITED ; // (1 currently) nv = 2 ; xu_ocean = 3600 ; yu_ocean = 2700 ; st_ocean = 50 ; st_edges_ocean = 51 ; sw_edges_ocean = 51 ; sw_ocean = 50 ; variables: double xt_ocean(xt_ocean) ; xt_ocean:long_name = "tcell longitude" ; xt_ocean:units = "degrees_E" ; xt_ocean:cartesian_axis = "X" ; double yt_ocean(yt_ocean) ; yt_ocean:long_name = "tcell latitude" ; yt_ocean:units = "degrees_N" ; yt_ocean:cartesian_axis = "Y" ; double time(time) ; time:long_name = "time" ; time:units = "days since 0001-01-01 00:00:00" ; time:cartesian_axis = "T" ; time:calendar_type = "JULIAN" ; time:calendar = "JULIAN" ; time:bounds = "time_bounds" ; double nv(nv) ; nv:long_name = "vertex number" ; nv:units = "none" ; nv:cartesian_axis = "N" ; double xu_ocean(xu_ocean) ; xu_ocean:long_name = "ucell longitude" ; xu_ocean:units = "degrees_E" ; xu_ocean:cartesian_axis = "X" ; double yu_ocean(yu_ocean) ; yu_ocean:long_name = "ucell latitude" ; yu_ocean:units = "degrees_N" ; yu_ocean:cartesian_axis = "Y" ; double st_ocean(st_ocean) ; st_ocean:long_name = "tcell zstar depth" ; st_ocean:units = "meters" ; st_ocean:cartesian_axis = "Z" ; st_ocean:positive = "down" ; st_ocean:edges = "st_edges_ocean" ; double st_edges_ocean(st_edges_ocean) ; st_edges_ocean:long_name = "tcell zstar depth edges" ; st_edges_ocean:units = "meters" ; st_edges_ocean:cartesian_axis = "Z" ; st_edges_ocean:positive = "down" ; double sw_edges_ocean(sw_edges_ocean) ; sw_edges_ocean:long_name = "ucell zstar depth edges" ; sw_edges_ocean:units = "meters" ; sw_edges_ocean:cartesian_axis = "Z" ; sw_edges_ocean:positive = "down" ; float geolon_t(yt_ocean, xt_ocean) ; geolon_t:long_name = "tracer longitude" ; geolon_t:units = "degrees_E" ; geolon_t:valid_range = -281.f, 361.f ; geolon_t:missing_value = 1.e+20f ; geolon_t:_FillValue = 1.e+20f ; geolon_t:cell_methods = "time: point" ; geolon_t:coordinates = "geolon_t geolat_t" ; float geolat_t(yt_ocean, xt_ocean) ; geolat_t:long_name = "tracer latitude" ; geolat_t:units = "degrees_N" ; geolat_t:valid_range = -91.f, 91.f ; geolat_t:missing_value = 1.e+20f ; geolat_t:_FillValue = 1.e+20f ; geolat_t:cell_methods = "time: point" ; geolat_t:coordinates = "geolon_t geolat_t" ; float geolon_c(yu_ocean, xu_ocean) ; geolon_c:long_name = "uv longitude" ; geolon_c:units = "degrees_E" ; geolon_c:valid_range = -281.f, 361.f ; geolon_c:missing_value = 1.e+20f ; geolon_c:_FillValue = 1.e+20f ; geolon_c:cell_methods = "time: point" ; geolon_c:coordinates = "geolon_c geolat_c" ; float geolat_c(yu_ocean, xu_ocean) ; geolat_c:long_name = "uv latitude" ; geolat_c:units = "degrees_N" ; geolat_c:valid_range = -91.f, 91.f ; geolat_c:missing_value = 1.e+20f ; geolat_c:_FillValue = 1.e+20f ; geolat_c:cell_methods = "time: point" ; geolat_c:coordinates = "geolon_c geolat_c" ; float temp(time, st_ocean, yt_ocean, xt_ocean) ; temp:long_name = "Potential temperature" ; temp:units = "degrees C" ; temp:valid_range = -10.f, 500.f ; temp:missing_value = -1.e+20f ; temp:_FillValue = -1.e+20f ; temp:cell_methods = "time: mean" ; temp:time_avg_info = "average_T1,average_T2,average_DT" ; temp:coordinates = "geolon_t geolat_t" ; temp:standard_name = "sea_water_potential_temperature" ; double time_bounds(time, nv) ; time_bounds:long_name = "time axis boundaries" ; time_bounds:units = "days" ; time_bounds:missing_value = 1.e+20 ; time_bounds:_FillValue = 1.e+20 ; float salt(time, st_ocean, yt_ocean, xt_ocean) ; salt:long_name = "Practical Salinity" ; salt:units = "psu" ; salt:valid_range = -10.f, 100.f ; salt:missing_value = -1.e+20f ; salt:_FillValue = -1.e+20f ; salt:cell_methods = "time: mean" ; salt:time_avg_info = "average_T1,average_T2,average_DT" ; salt:coordinates = "geolon_t geolat_t" ; salt:standard_name = "sea_water_salinity" ; float u(time, st_ocean, yu_ocean, xu_ocean) ; u:long_name = "i-current" ; u:units = "m/sec" ; u:valid_range = -10.f, 10.f ; u:missing_value = -1.e+20f ; u:_FillValue = -1.e+20f ; u:cell_methods = "time: mean" ; u:time_avg_info = "average_T1,average_T2,average_DT" ; u:coordinates = "geolon_c geolat_c" ; u:standard_name = "sea_water_x_velocity" ; float v(time, st_ocean, yu_ocean, xu_ocean) ; v:long_name = "j-current" ; v:units = "m/sec" ; v:valid_range = -10.f, 10.f ; v:missing_value = -1.e+20f ; v:_FillValue = -1.e+20f ; v:cell_methods = "time: mean" ; v:time_avg_info = "average_T1,average_T2,average_DT" ; v:coordinates = "geolon_c geolat_c" ; v:standard_name = "sea_water_y_velocity" ; float pot_rho_0(time, st_ocean, yt_ocean, xt_ocean) ; pot_rho_0:long_name = "potential density referenced to 0 dbar" ; pot_rho_0:units = "kg/m^3" ; pot_rho_0:valid_range = -10.f, 100000.f ; pot_rho_0:missing_value = -1.e+20f ; pot_rho_0:_FillValue = -1.e+20f ; pot_rho_0:cell_methods = "time: mean" ; pot_rho_0:time_avg_info = "average_T1,average_T2,average_DT" ; pot_rho_0:coordinates = "geolon_t geolat_t" ; pot_rho_0:standard_name = "sea_water_potential_density" ; float ty_trans(time, st_ocean, yu_ocean, xt_ocean) ; ty_trans:long_name = "T-cell j-mass transport" ; ty_trans:units = "Sv (10^9 kg/s)" ; ty_trans:valid_range = -1.e+20f, 1.e+20f ; ty_trans:missing_value = -1.e+20f ; ty_trans:_FillValue = -1.e+20f ; ty_trans:cell_methods = "time: mean" ; ty_trans:time_avg_info = "average_T1,average_T2,average_DT" ; ty_trans:coordinates = "geolon_t geolat_c" ; ty_trans:standard_name = "ocean_y_mass_transport" ; float eta_t(time, yt_ocean, xt_ocean) ; eta_t:long_name = "surface height on T cells [Boussinesq (volume conserving) model]" ; eta_t:units = "meter" ; eta_t:valid_range = -1000.f, 1000.f ; eta_t:missing_value = -1.e+20f ; eta_t:_FillValue = -1.e+20f ; eta_t:cell_methods = "time: mean" ; eta_t:time_avg_info = "average_T1,average_T2,average_DT" ; eta_t:coordinates = "geolon_t geolat_t" ; float eta_u(time, yu_ocean, xu_ocean) ; eta_u:long_name = "surface height on U cells" ; eta_u:units = "meter" ; eta_u:valid_range = -1000.f, 1000.f ; eta_u:missing_value = -1.e+20f ; eta_u:_FillValue = -1.e+20f ; eta_u:cell_methods = "time: mean" ; eta_u:time_avg_info = "average_T1,average_T2,average_DT" ; eta_u:coordinates = "geolon_c geolat_c" ; float frazil_2d(time, yt_ocean, xt_ocean) ; frazil_2d:long_name = "ocn frazil heat flux over time step" ; frazil_2d:units = "W/m^2" ; frazil_2d:valid_range = -1.e+10f, 1.e+10f ; frazil_2d:missing_value = -1.e+20f ; frazil_2d:_FillValue = -1.e+20f ; frazil_2d:cell_methods = "time: mean" ; frazil_2d:time_avg_info = "average_T1,average_T2,average_DT" ; frazil_2d:coordinates = "geolon_t geolat_t" ; float hblt(time, yt_ocean, xt_ocean) ; hblt:long_name = "T-cell boundary layer depth from KPP" ; hblt:units = "m" ; hblt:valid_range = -100000.f, 1000000.f ; hblt:missing_value = -1.e+20f ; hblt:_FillValue = -1.e+20f ; hblt:cell_methods = "time: mean" ; hblt:time_avg_info = "average_T1,average_T2,average_DT" ; hblt:coordinates = "geolon_t geolat_t" ; hblt:standard_name = "ocean_mixed_layer_thickness_defined_by_mixing_scheme" ; float mld(time, yt_ocean, xt_ocean) ; mld:long_name = "mixed layer depth determined by density criteria" ; mld:units = "m" ; mld:valid_range = 0.f, 1000000.f ; mld:missing_value = -1.e+20f ; mld:_FillValue = -1.e+20f ; mld:cell_methods = "time: mean" ; mld:time_avg_info = "average_T1,average_T2,average_DT" ; mld:coordinates = "geolon_t geolat_t" ; mld:standard_name = "ocean_mixed_layer_thickness_defined_by_sigma_t" ; float mld_dtheta(time, yt_ocean, xt_ocean) ; mld_dtheta:long_name = "mixed layer depth determined by temperature criteria" ; mld_dtheta:units = "m" ; mld_dtheta:valid_range = 0.f, 1000000.f ; mld_dtheta:missing_value = -1.e+20f ; mld_dtheta:_FillValue = -1.e+20f ; mld_dtheta:cell_methods = "time: mean" ; mld_dtheta:time_avg_info = "average_T1,average_T2,average_DT" ; mld_dtheta:coordinates = "geolon_t geolat_t" ; float net_sfc_heating(time, yt_ocean, xt_ocean) ; net_sfc_heating:long_name = "surface ocean heat flux coming through coupler and mass transfer" ; net_sfc_heating:units = "Watts/m^2" ; net_sfc_heating:valid_range = -10000.f, 10000.f ; net_sfc_heating:missing_value = -1.e+20f ; net_sfc_heating:_FillValue = -1.e+20f ; net_sfc_heating:cell_methods = "time: mean" ; net_sfc_heating:time_avg_info = "average_T1,average_T2,average_DT" ; net_sfc_heating:coordinates = "geolon_t geolat_t" ; float pme_river(time, yt_ocean, xt_ocean) ; pme_river:long_name = "mass flux of precip-evap+river via sbc (liquid, frozen, evaporation)" ; pme_river:units = "(kg/m^3)(m/sec)" ; pme_river:valid_range = -1000000.f, 1000000.f ; pme_river:missing_value = -1.e+20f ; pme_river:_FillValue = -1.e+20f ; pme_river:cell_methods = "time: mean" ; pme_river:time_avg_info = "average_T1,average_T2,average_DT" ; pme_river:coordinates = "geolon_t geolat_t" ; pme_river:standard_name = "water_flux_into_sea_water" ; float river(time, yt_ocean, xt_ocean) ; river:long_name = "mass flux of river (runoff + calving) entering ocean" ; river:units = "(kg/m^3)(m/sec)" ; river:valid_range = -1000000.f, 1000000.f ; river:missing_value = -1.e+20f ; river:_FillValue = -1.e+20f ; river:cell_methods = "time: mean" ; river:time_avg_info = "average_T1,average_T2,average_DT" ; river:coordinates = "geolon_t geolat_t" ; float salt_int_rhodz(time, yt_ocean, xt_ocean) ; salt_int_rhodz:long_name = "vertical sum of Practical Salinity * rho_dzt" ; salt_int_rhodz:units = "psu(kg/m^3)m" ; salt_int_rhodz:valid_range = -1.e+20f, 1.e+20f ; salt_int_rhodz:missing_value = -1.e+20f ; salt_int_rhodz:_FillValue = -1.e+20f ; salt_int_rhodz:cell_methods = "time: mean" ; salt_int_rhodz:time_avg_info = "average_T1,average_T2,average_DT" ; salt_int_rhodz:coordinates = "geolon_t geolat_t" ; float sea_level(time, yt_ocean, xt_ocean) ; sea_level:long_name = "effective sea level (eta_t + patm/(rho0g)) on T cells" ; sea_level:units = "meter" ; sea_level:valid_range = -1000.f, 1000.f ; sea_level:missing_value = -1.e+20f ; sea_level:_FillValue = -1.e+20f ; sea_level:cell_methods = "time: mean" ; sea_level:time_avg_info = "average_T1,average_T2,average_DT" ; sea_level:coordinates = "geolon_t geolat_t" ; sea_level:standard_name = "sea_surface_height_above_geoid" ; float sea_levelsq(time, yt_ocean, xt_ocean) ; sea_levelsq:long_name = "square of effective sea level (eta_t + patm/(rho0g)) on T cells" ; sea_levelsq:units = "m^2" ; sea_levelsq:valid_range = -1000.f, 1000.f ; sea_levelsq:missing_value = -1.e+20f ; sea_levelsq:_FillValue = -1.e+20f ; sea_levelsq:cell_methods = "time: mean" ; sea_levelsq:time_avg_info = "average_T1,average_T2,average_DT" ; sea_levelsq:coordinates = "geolon_t geolat_t" ; sea_levelsq:standard_name = "square_of_sea_surface_height_above_geoid" ; float sfc_hflux_coupler(time, yt_ocean, xt_ocean) ; sfc_hflux_coupler:long_name = "surface heat flux coming through coupler" ; sfc_hflux_coupler:units = "Watts/m^2" ; sfc_hflux_coupler:valid_range = -10000.f, 10000.f ; sfc_hflux_coupler:missing_value = -1.e+20f ; sfc_hflux_coupler:_FillValue = -1.e+20f ; sfc_hflux_coupler:cell_methods = "time: mean" ; sfc_hflux_coupler:time_avg_info = "average_T1,average_T2,average_DT" ; sfc_hflux_coupler:coordinates = "geolon_t geolat_t" ; double sw_ocean(sw_ocean) ; sw_ocean:long_name = "ucell zstar depth" ; sw_ocean:units = "meters" ; sw_ocean:cartesian_axis = "Z" ; sw_ocean:positive = "down" ; sw_ocean:edges = "sw_edges_ocean" ; float tau_x(time, yu_ocean, xu_ocean) ; tau_x:long_name = "i-directed wind stress forcing u-velocity" ; tau_x:units = "N/m^2" ; tau_x:valid_range = -10.f, 10.f ; tau_x:missing_value = -1.e+20f ; tau_x:_FillValue = -1.e+20f ; tau_x:cell_methods = "time: mean" ; tau_x:time_avg_info = "average_T1,average_T2,average_DT" ; tau_x:coordinates = "geolon_c geolat_c" ; tau_x:standard_name = "surface_downward_x_stress" ; float tau_y(time, yu_ocean, xu_ocean) ; tau_y:long_name = "j-directed wind stress forcing v-velocity" ; tau_y:units = "N/m^2" ; tau_y:valid_range = -10.f, 10.f ; tau_y:missing_value = -1.e+20f ; tau_y:_FillValue = -1.e+20f ; tau_y:cell_methods = "time: mean" ; tau_y:time_avg_info = "average_T1,average_T2,average_DT" ; tau_y:coordinates = "geolon_c geolat_c" ; tau_y:standard_name = "surface_downward_y_stress" ; float temp_int_rhodz(time, yt_ocean, xt_ocean) ; temp_int_rhodz:long_name = "vertical sum of Potential temperature * rho_dzt" ; temp_int_rhodz:units = "deg_C(kg/m^3)m" ; temp_int_rhodz:valid_range = -1.e+20f, 1.e+20f ; temp_int_rhodz:missing_value = -1.e+20f ; temp_int_rhodz:_FillValue = -1.e+20f ; temp_int_rhodz:cell_methods = "time: mean" ; temp_int_rhodz:time_avg_info = "average_T1,average_T2,average_DT" ; temp_int_rhodz:coordinates = "geolon_t geolat_t" ; float wt(time, sw_ocean, yt_ocean, xt_ocean) ; wt:long_name = "dia-surface velocity T-points" ; wt:units = "m/sec" ; wt:valid_range = -100000.f, 100000.f ; wt:missing_value = -1.e+20f ; wt:_FillValue = -1.e+20f ; wt:cell_methods = "time: mean" ; wt:time_avg_info = "average_T1,average_T2,average_DT" ; wt:coordinates = "geolon_t geolat_t" ;

// global attributes: :filename = "01980101.ocean.nc" ; :title = "CM2.6_miniBling" ; :grid_type = "mosaic" ; :grid_tile = "1" ; :history = "Tue Feb 25 16:32:17 2014: ncks --64bit --hdr_pad 15000 -A ocean.0198.ann2d.nc ocean.0198.ann.nc\n", "Tue Feb 25 16:23:35 2014: ncks --64bit --hdr_pad 15000 -A frazil_2d.nc ocean.0198.ann2d.nc\n", "Tue Feb 25 16:23:32 2014: ncra -O -v nv,time_bounds,geolat_c,geolat_t,geolon_c,geolon_t,st_edges_ocean,sw_edges_ocean,frazil_2d 01980101.ocean.nc frazil_2d.nc" ; :nco_openmp_thread_number = 1 ; :NCO = "4.1.0" ; } ```

I am unsure if I am allowed to share them publicly, is there another way to diagnose what is going on?

Thanks a lot!

Can you share an example file?

Note that there are some performance improvements for open_mfdataset in the next version of xarray, which you could test out with the release candidate if you like.