issue_comments

I tried your suggestion and still ran into a memory allocation error, but it sounds like you're onto something. I also found this other thread about reindex causing memory allocation errors but it doesn't look like a solution was discovered there either. https://github.com/pydata/xarray/issues/2745

I found a functional workaround is to chunk by one of the spatial dimensions instead. I'd still like to know why the code above fails though. I'm assuming there's a scheduled task with dask the occurs before to_netcdf but I haven't been able to figure out what that is.

python ds = xr.open_dataset(fldr_in_grib, engine="cfgrib", chunks={"latitude":875}, backend_kwargs={'indexpath': ''})

I cross posted this as a dask issue and on stack overflow. I learned that "dask will often have as many chunks in memory as twice the number of active threads" (best practices with dask arrays) and including dask.config.set(scheduler='synchronous') which forces single thread computation (dask scheduling) resulted in the behavior I expected which is memory usage fluctuations that are roughly the magnitude of the chunk size.

Thanks, @andersy005. I think that xr.save_mfdataset() could certainly be helpful in my workflow but unfortunately, I have to consolidate these data from a netcdf for each 2-minute timestep to a netcdf for each day, and it sounds like there's no way around that bottleneck. I've come across suggestions to save the dataset to a zarr group and then export as a netcdf, so I'm going to give that a shot.

I realized that the reason I wasn't seeing what I was expecting was because of the dataset, not the plotting function. There shouldn't be any 0s in the rainrate variable.

I can't believe I forgot the asterisk!!! Thank you for catching that.

I just edited the original post to show that xr.open_dataset works. I hope that is adequate to show that the issue is with xarray and not netcdf4.Dataset.