An issue already exists on the dask issue tracker https://github.com/dask/dask/issues/5679

This seems to be an upstream issue:

da.data.std(axis=0).compute()

I will close shortly, apologizes

I've got issues reproducing this issue, so closing for now.

argh ... saved_on_disk.nc has been removed.

I was unfortunately not able to create a proper list without breaking tests and have exhausted my time trying to figure it out. If you know how to do that I am more than happy to follow your advice.

ok, tests are passing.

if anybody has a clue on how to fix tests this would be welcome. thx

Thanks for this speedy reply @rabernat !

Improving docs is still within my reach (I hope) and I will give it a shot. Could this improvement in the document take place in the description of the encoding parameter of xarray.Dataset.to_zarr?

you're welcome, I hope one day I'll be to make a more worthy contribution to this great library !

I would feel bad claiming credit for such a small contribution ... it's not even solving my pb which I should probably better document on the issue tracker.

Ok, let me know if you need me to do anything else.

Argh, my bad. This implementation does not solve #1889.

hehe. I cannot reopen the issue apparently

I reopen this issue as it came across my road again when generating figures on a dask.Distributed LocalCluster. I just open a PR suggesting a change that solves the issue in my situation.

I'll close this for now as there doesn't seem to be other ideas about this

This sounds like method 1 (with dask delayed) to me. There may be no faster option, thanks for giving it a thought @fujiisoup

thanks a lot @fujiisoup, your suggestion does help getting rid of the necessity to build the ds['_y'] variable. Here is the updated apply_ufunc solution: ``` x = np.arange(10100) y = np.arange(20100)

ds = xr.Dataset(coords={'x': x, 'y': y})

ds = ds.chunk({'x': 1, 'y':1}) # does not change anything

let's say each experiment outputs 5 statistical diagnostics

Nstats = 5 some_exp = lambda x, y: np.ones((Nstats,))

out = xr.apply_ufunc(some_exp, ds.x, ds.y, dask='parallelized', vectorize=True, output_dtypes=[float], output_sizes={'stats': Nstats}, output_core_dims=[['stats']]) ``` An inspection of the dask dashboard indicates that the computation is not distributed among workers though. How could I make sure this happens?

sorry closed by accident

thanks to you for the fix @TomNicholas !

I have ended up using apply_ufunc at several occasions and have developed a love/hate relationship with it. Often it turned out to be the simplest and most powerful option ... once I figured how to use it.

So thumbs up for an improved documentation.

Undertaking this task seems like a daunting one to me however, mostly because there are many different ways of using apply_ufunc I am not familiar with. Maybe it's the case for other users as well ...?

If this is the case, shouldn't we 1/ gather clean versions of our examples in a temporary place, 2/ sort these examples, and 3/ consider pushing it as a doc ?

ds = xr.open_dataset(grid_dir_nc+'Depth.nc', chunks={'face':1}, engine='h5netcdf') % time print(ds.Depth.mean().values) leads to: CPU times: user 824 ms, sys: 50.4 ms, total: 875 ms Wall time: 11.3 s and ds = xr.open_dataset(grid_dir_nc+'Depth.nc', chunks={'face':1}).load() % time print(ds.Depth.mean().values) to CPU times: user 61.9 ms, sys: 22.4 ms, total: 84.3 ms Wall time: 76.2 ms much better ... thanks a lot for these two solutions.

additional information about the file:

``` (equinox) [pontea@visu01 LLC4320]$ ncdump -sh grid_nc/Depth.nc netcdf Depth { dimensions: i = 4320 ; j = 4320 ; face = 13 ; variables: int64 i(i) ; i:standard_name = "x_grid_index" ; i:axis = "X" ; i:long_name = "x-dimension of the t grid" ; i:swap_dim = "XC" ; i:_Storage = "contiguous" ; i:_Endianness = "little" ; int64 j(j) ; j:standard_name = "y_grid_index" ; j:axis = "Y" ; j:long_name = "y-dimension of the t grid" ; j:swap_dim = "YC" ; j:_Storage = "contiguous" ; j:_Endianness = "little" ; int64 face(face) ; face:standard_name = "face_index" ; face:_Storage = "contiguous" ; face:_Endianness = "little" ; float Depth(face, j, i) ; Depth:_FillValue = NaNf ; Depth:standard_name = "ocean_depth" ; Depth:long_name = "ocean depth" ; Depth:units = "m" ; Depth:coordinate = "XC YC" ; Depth:_Storage = "chunked" ; Depth:_ChunkSizes = 1, 480, 480 ; Depth:_Endianness = "little" ;

// global attributes: :_SuperblockVersion = 0 ; :_IsNetcdf4 = 1 ; :_Format = "netCDF-4" ; } ```

Could you please be more specific about where this is done for netCDF?

Note that there is also a fix for case 2 that is simply del ds['v'].encoding['chunks'] prior to data storage.

With the same case, I have another error message which may reflect the same issue (or not), maybe you can tell me. The error message is different which is the reason I am posting this.

Starting from the same dataset: nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.t*np.cos(np.pi/180./100*ds.x)*np.cos(np.pi/180./50*ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2}) ds.to_zarr('data.zarr', mode='w')

Case 1 works fine: ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w')

Case 2 breaks: ds = xr.open_zarr('data.zarr') ds = ds.chunk({'t': nt, 'x': nx/4, 'y': ny/4}) ds.to_zarr('data_rechunked.zarr', mode='w') with the following error message: .... NotImplementedError: Specified zarr chunks (1, 16, 16) would overlap multiple dask chunks ((64,), (8, 8, 8, 8), (8, 8, 8, 8)). This is not implemented in xarray yet. Consider rechunking the data using `chunk()` or specifying different chunks in encoding.

thanks for the workaround suggestion. Apparently you also need to delete chunks for the t singleton coordinate though. The workaround looks at the end like: ds = xr.open_zarr('data.zarr') del ds['v'].encoding['chunks'] del ds['t'].encoding['chunks'] ds.isel(t=0).to_zarr('data_t0.zarr', mode='w') Any idea about how serious this is and/or where it's coming from?

it turned out I had exceeded my quota on the storage.

Sorry about the noise.

I will look into zarr though.

Ok, thanks, do you have a piece of code with a thread-lock that I could get inspiration from?

thanks, got it. I tried the suggested fix but it did not work. Unfortunately I don't have more time on the subject for a couple of days. I'll keep you posted when I can sort things out.

Hi,

Yes why not even though I am not too familiar with the process.

I am not even able to properly install the library so far ... python setup.py install creates the following library: /home1/datahome/aponte/.miniconda3/envs/pangeo/lib/python3.6/site-packages/xarray-0.10.0rc1_2_gf83361c-py3.6.egg/ I am surely doing something wrong, I'd like to have xarray.egg.

Along time_counter

Thanks for your answer. You'll find below an output from ncdump in order to answer the part about data arrangement (I hope ...) Otherwise, we have:

dask 0.8.2 py27_0 defaults xarray 0.7.2 py27_0 defaults netcdf4 1.1.1 np18py27_0 defaults

Looking forward for any suggestions. cheers aurelien

``` netcdf NATL60-MJM155_y2008m01d09.5d_gridT { dimensions: x = 5422 ; y = 3454 ; deptht = 300 ; time_counter = UNLIMITED ; // (1 currently) time_bounds = 2 ; variables: float nav_lat(y, x) ; nav_lat:axis = "Y" ; nav_lat:standard_name = "latitude" ; nav_lat:long_name = "Latitude" ; nav_lat:units = "degrees_north" ; nav_lat:nav_model = "grid_T" ; nav_lat:_Storage = "chunked" ; nav_lat:_ChunkSizes = 12, 5422 ; float nav_lon(y, x) ; nav_lon:axis = "X" ; nav_lon:standard_name = "longitude" ; nav_lon:long_name = "Longitude" ; nav_lon:units = "degrees_east" ; nav_lon:nav_model = "grid_T" ; nav_lon:_Storage = "chunked" ; nav_lon:_ChunkSizes = 12, 5422 ; float deptht(deptht) ; deptht:axis = "Z" ; deptht:long_name = "Vertical T levels" ; deptht:units = "m" ; deptht:positive = "down" ; deptht:_Storage = "chunked" ; deptht:_ChunkSizes = 300 ; float votemper(time_counter, deptht, y, x) ; votemper:long_name = "temperature" ; votemper:units = "degC" ; votemper:online_operation = "average" ; votemper:interval_operation = "40s" ; votemper:interval_write = "5d" ; votemper:_FillValue = 0.f ; votemper:missing_value = 0.f ; votemper:coordinates = "time_centered deptht nav_lon nav_lat" ; votemper:_Storage = "chunked" ; votemper:_ChunkSizes = 1, 1, 12, 5422 ; votemper:_DeflateLevel = 1 ; double time_centered(time_counter) ; time_centered:standard_name = "time" ; time_centered:long_name = "Time axis" ; time_centered:title = "Time" ; time_centered:calendar = "gregorian" ; time_centered:units = "seconds since 1958-01-01 00:00:00" ; time_centered:time_origin = "1958-01-01 00:00:00" ; time_centered:bounds = "time_centered_bounds" ; time_centered:_Storage = "chunked" ; time_centered:_ChunkSizes = 1 ; double time_centered_bounds(time_counter, time_bounds) ; time_centered_bounds:_Storage = "chunked" ; time_centered_bounds:_ChunkSizes = 1, 2 ; double time_counter(time_counter) ; time_counter:axis = "T" ; time_counter:standard_name = "time" ; time_counter:long_name = "Time axis" ; time_counter:title = "Time" ; time_counter:calendar = "gregorian" ; time_counter:units = "seconds since 1958-01-01 00:00:00" ; time_counter:time_origin = "1958-01-01 00:00:00" ; time_counter:bounds = "time_counter_bounds" ; time_counter:_Storage = "chunked" ; time_counter:_ChunkSizes = 1 ; double time_counter_bounds(time_counter, time_bounds) ; time_counter_bounds:_Storage = "chunked" ; time_counter_bounds:_ChunkSizes = 1, 2 ; float vosaline(time_counter, deptht, y, x) ; vosaline:long_name = "salinity" ; vosaline:units = "psu" ; vosaline:online_operation = "average" ; vosaline:interval_operation = "40s" ; vosaline:interval_write = "5d" ; vosaline:_FillValue = 0.f ; vosaline:missing_value = 0.f ; vosaline:coordinates = "time_centered deptht nav_lon nav_lat" ; vosaline:_Storage = "chunked" ; vosaline:_ChunkSizes = 1, 1, 12, 5422 ; vosaline:_DeflateLevel = 1 ; float sossheig(time_counter, y, x) ; sossheig:long_name = "sea surface height" ; sossheig:units = "m" ; sossheig:online_operation = "average" ; sossheig:interval_operation = "40s" ; sossheig:interval_write = "5d" ; sossheig:_FillValue = 0.f ; sossheig:missing_value = 0.f ; sossheig:coordinates = "time_centered nav_lon nav_lat" ; sossheig:_Storage = "chunked" ; sossheig:_ChunkSizes = 1, 12, 5422 ;

// global attributes: :description = "ocean T grid variables" ; :conventions = "CF-1.1" ; :production = "An IPSL model" ; :start_date = 20040101 ; :output_frequency = "5d" ; :CONFIG = "NATL60" ; :CASE = "MJM155" ; :_Format = "netCDF-4" ; } ```