issue_comments

Alternatively, it could make sense to change pickle upstream in NumPy to special case arrays with a stride of 0 along some dimension differently.

@crusaderky Yes, I think it could be reasonable to unify array types when you call broadcast() or align(), as either as optional behavior or by changing the default.

If your scalar array is the result of an expensive dask calculation, this also might be a good use case for dask's new .persist() method (https://github.com/dask/dask/issues/1908), which we could add to xarray as an alternative to .compute().

Actually, I very much still am facing the problem. The biggest issue is now when I need to invoke xarray.broadcast. In my use case, I'm broadcasting together

• a scalar array with numpy backend, shape=(), chunks=None
• a 1D array with dask backend, shape=(2**19,), chunks=(2**15,)

What broadcast does is transform the scalar array to a numpy array of 2**19 elements. This is actually a view on the original 0D array, so it's got negligible RAM requirements. But after pickling and unpickling, it's become a real 2**19 elements array. Add up a few hundreds of them, and I am facing GBs of wasted RAM.

A solution would be to change broadcast() to convert to dask before broadcasting, and then broadcast directly to the proper chunk size.

I think this is fixed about as well as we can hope given how pickle works for NumPy by https://github.com/pydata/xarray/pull/1128.

So I'm closing this now, but feel free to open another issue for any follow-up concerns.

Confirmed that #1017 fixes my specific issue, thanks! Leaving the ticket open as other people (particularly those that work on large arrays without dask) will still be affected.

I answered the StackOverflow question: https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling/40247761#40247761

This was a tricky puzzle to figure out!

@crusaderky right, I see. All those views are in the same pickle object, and so shouldn't be duplicated. That is frustrating.

As per @shoyer, the easiest way is to just not have the data in the first place. So not needing indexes at all should solve your case.

@maximilianr, if you pickle 2 plain python objects A and B together, and one of the attributes of B is a reference to A, A does not get duplicated.

In this case there must be some specific getstate code to prevent this and/or something with the C implementation of the class

If I'm understanding you correctly @crusaderky, I think this is a tough problem, and one much broader than xarray. When pickling something with a reference, do you want to save the object, or the reference? If you pickle the reference, how can you guarantee to have the object available when unpickling? How would you codify the reference (memory location?)?

Is that right? Or am I misunderstanding your problem?

On this narrow case, I think not having indexes at all should solve this, though

The plan is stop making default indexes with np.arange. See https://github.com/pydata/xarray/pull/1017, which is my top priority for the next major release.

I'm not confident that your work around will work properly. At the very least, you should check strides as well. Otherwise get_base(array[::-1]) would return array.

If it would really help, I'm open to making Variable(dims, array) reuse the same numpy array instead of creating a view (see as_compatible_data).