issue_comments

Happy to make any changes required!!

Great, will prepare a PR! I second @fmaussion on keeping the auto colorbar, just makes life easy!

The DataArrays we use are a thin wrapper over xarray's to allow conversion into desired units. If we used Dataset, then we lose this functionality unless we subclass Dataset too. Overall, using a simple dictionary suits our purposes better.

Thanks, Ryan. This is for CliMT, where the model arrays are DataArrays, so it would not make sense to use a Dataset. I think option 1 will make more sense.

Also, I recall this is new functionality. What minimum version must I use to use 2D coordinates? That is, once I get the right syntax, that is ;)

Not sure if this helps, but I did a %%timeit on both versions. For daily climatology, the numbers are: CPU times: user 1h 21min 8s, sys: 6h 17min 39s, total: 7h 38min 47s Wall time: 20min 34s

For the 6 hourly thing, CPU times: user 5h 5min 6s, sys: 1d 2h 19min 45s, total: 1d 7h 24min 51s Wall time: 1h 31min 40s

It takes around 4x more time, which makes sense because there are 4x more groups. The ratio of user to system time is more or less constant, so nothing untoward seems to be happening in between the two runs.

I think it is just good to remember that the time to use scales linearly with the number of groups. I guess this is what @shoyer was talking about when he mentioned that since grouping is done within xarray, the dask graph grows, making things slower.

Thanks again!

@shoyer If I increase the size of the longitude chunk anymore, it will almost like using no chunking at all. I guess this dataset is a corner case. I will try increasing doubling that value and see what happens. I hadn't realised that doing a groupby would also reduce the effective chunk size, thanks for pointing that out.

I'm using dask without distributed as of now, is there still some way to do the benchmark? I would be more than happy to run it.

@rabernat I would definitely favour a cloud based sandbox to try these things out. What would be the stumbling block towards actually setting it up? I have had some recent experience setting up jupyterhub, I can help set that up so that notebooks can be used easily in such an environment.

Hello Stephan,

The shape of the full data, if I read from within xarray, is (time, level, lat, lon), with level=60, lat=41, lon=480. time is 4*365*7 ~ 10000.

I am chunking only along longitude, using lon=100. I previously chunked along time, but that used too much memory (~45GB out of 128 GB) since the data is split into one file per month, and reading annual data would require reading many files into memory.

Superficially, I would think that both of the above would take similar amounts of time. In fact, calculating a daily climatology also requires grouping the four 6 hourly data points into a single day as well, which seems to be more complicated. However, it seems to run faster!

Thanks, Joy

Playing around with things sounds like much more fun :) I can see how this will be useful, will start thinking of some test cases to code.

Thanks. how big of an endeavour is this? I see some free time from 2-3rd week of Jan, and I could maybe contribute towards making this happen.

@shoyer: does this also work with dask.distributed? The doc seems to only mention a thread pool.

did not know about that, thanks!!

Oh, not really! thanks for the fix :)

Just repeating your example, but with

python z.plot.contour(ax=ax1, norm=None, **kw)

gives the expected

for some reason, the norm is causing an issue.

Yes, I had encountered that as well, but did not realise it was due to the colorbar... I normally don't use the colorbar. Thanks for the example!

Yes, that makes sense. quite the corner case! works perfectly now, thanks again.

You were right, my chunk sizes were too large. It did not matter how many threads dask used either (4 vs. 8). The I/O component is still high, but that is also because I'm writing the final computed DataArray to disk.

Thanks!

Hello, I ran it with the dask profiler, and I looked at the top output disaggregated by core. It does seem to use multiple cores, but it seems to be using 8 threads when I looked at prof.visualize() (hyperthreading :P) and I feel this is killing performance.

How can I control how many threads to use?

Thanks, Joy

I was trying to read ERA-Interim data, calculate anomalies using ds = ds - ds.mean(dim='longitude'), and similar operations along the time axis. Are such operations restricted to single cores?

Just multiplying two datasets (u*v) seems to be faster, though top shows two cores being used (I have 4 physical cores).

TIA, Joy

Thanks! Will extend my code in this fashion.

Thanks. But that really kills my machine, even though I have 12 GB of RAM.

What I finally ended up doing is slicing the initial dataset created from one nc file to access the level+variable that I wanted. This gives me a DataArray object which I then xray.concat() with similar objects created from other variables.

No, not really. each file contains one year of data for four variables, and I have 35 files (1979-...)

I tried Dataset.merge as you suggested, but it says conflicting value for variable time, which I guess is what you would expect.

Can xray modify the nc file to make the time dimension unlimited? then I could simply use something like MFDataset...

TIA, Joy