id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 99026442,MDU6SXNzdWU5OTAyNjQ0Mg==,516,Wall time much greater than CPU time,3688009,closed,0,,,5,2015-08-04T18:10:50Z,2016-12-29T01:09:54Z,2016-12-29T01:09:54Z,NONE,,,,"I have a very large data set. It is multiple files, and each file is internally compressed. I have it all locally downloaded unto my external hard drive (the data is so large, this is the only place I can currently hold it, and I think this is some effect, but I am not sure if it is the entire effect. It is a USB 3 external though, which helps. Not ideal either way). By loading in my data with `xray.open_mfdataset(filename, chunks={'time':365, 'lat':180, 'lon':360})` and doing a simple call along one of the dimensions, it takes: ``` %time data.tasmax[:, 360, 720].values CPU times: user 31.4 s, sys: 4.07 s, total: 35.4 s Wall time: 8min 38s ``` So, i decided that maybe the internal compression was really my limiting factor. So I saved my entire data set using ``` data.to_netcdf('nc_test.nc') ``` which resulted in a 268 Gb files, and it took between 12 and 16 hours to run. Now, I can load in that new dataset, using `xray.open_dataset` instead of `open_mfdataset`. So, i tried a few different loads to see if chunking helped the issue. ``` data = xray.open_mfdataset('/mnt/usb/CANESM2/tasmax*rcp45*Can*', chunks={'time':365, 'lat':180, 'lon':360}) new = xray.open_dataset('/mnt/usb/CANESM2/nasa_CANESM2_prediction.nc', chunks={'time':365, 'lat':180, 'lon':360}) new2 = xray.open_dataset('/mnt/usb/CANESM2/nasa_CANESM2_prediction.nc', chunks={'time':365, 'lat':360, 'lon':720}) new3 = xray.open_dataset('/mnt/usb/CANESM2/nasa_CANESM2_prediction.nc', chunks={'time':365, 'lat':720, 'lon':1440}) new4 = xray.open_dataset('/mnt/usb/CANESM2/nasa_CANESM2_prediction.nc', chunks={'time':34675, 'lat':720, 'lon':1440}) new5 = xray.open_mfdataset('/mnt/usb/CANESM2/nasa_CANESM2_prediction.nc', chunks={'time':365, 'lat':180, 'lon':360}) ``` and the resulting times (note that I change the index in each one, to avoid file caching): ``` %time data.tasmax[:, 360, 720].values CPU times: user 31.4 s, sys: 4.07 s, total: 35.4 s Wall time: 8min 38s %time new.tasmax[:, 360, 720].values CPU times: user 1.53 s, sys: 2.9 s, total: 4.43 s Wall time: 5min 35s %time new2.tasmax[:, 362, 721].values CPU times: user 817 ms, sys: 2.89 s, total: 3.71 s Wall time: 4min 7s %time new3.tasmax[:, 361, 720].values CPU times: user 987 ms, sys: 3.34 s, total: 4.33 s Wall time: 4min 17s %time new4.tasmax[:, 360, 720].values CPU times: user 713 ms, sys: 2.68 s, total: 3.4 s Wall time: 6min 5s %time new5.tasmax[:, 361, 720].values CPU times: user 1.25 s, sys: 2.79 s, total: 4.04 s Wall time: 5min 2s ``` The wall time is always greater than the CPU time (and CPU and Sys combined). Any insight? Can provide more info on request. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/516/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue