html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/1058#issuecomment-277549915,https://api.github.com/repos/pydata/xarray/issues/1058,277549915,MDEyOklzc3VlQ29tbWVudDI3NzU0OTkxNQ==,1217238,2017-02-05T21:13:41Z,2017-02-05T21:13:41Z,MEMBER,"Alternatively, it could make sense to change pickle upstream in NumPy to special case arrays with a stride of 0 along some dimension differently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-277549355,https://api.github.com/repos/pydata/xarray/issues/1058,277549355,MDEyOklzc3VlQ29tbWVudDI3NzU0OTM1NQ==,1217238,2017-02-05T21:06:19Z,2017-02-05T21:06:19Z,MEMBER,"@crusaderky Yes, I think it could be reasonable to unify array types when you call `broadcast()` or `align()`, as either as optional behavior or by changing the default.
If your scalar array is the result of an expensive dask calculation, this also might be a good use case for dask's new `.persist()` method (https://github.com/dask/dask/issues/1908), which we could add to xarray as an alternative to `.compute()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-277543644,https://api.github.com/repos/pydata/xarray/issues/1058,277543644,MDEyOklzc3VlQ29tbWVudDI3NzU0MzY0NA==,6213168,2017-02-05T19:44:33Z,2017-02-05T19:44:33Z,MEMBER,"Actually, I very much still am facing the problem.
The biggest issue is now when I need to invoke xarray.broadcast. In my use case, I'm broadcasting together
- a scalar array with numpy backend, shape=(), chunks=None
- a 1D array with dask backend, shape=(2\*\*19,), chunks=(2\*\*15,)
What broadcast does is transform the scalar array to a numpy array of 2\*\*19 elements. This is actually a view on the original 0D array, so it's got negligible RAM requirements. But after pickling and unpickling, it's become a real 2\*\*19 elements array. Add up a few hundreds of them, and I am facing GBs of wasted RAM.
A solution would be to change broadcast() to convert to dask *before* broadcasting, and then broadcast directly to the proper chunk size.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-273001734,https://api.github.com/repos/pydata/xarray/issues/1058,273001734,MDEyOklzc3VlQ29tbWVudDI3MzAwMTczNA==,1217238,2017-01-17T01:53:18Z,2017-01-17T01:53:18Z,MEMBER,"I think this is fixed about as well as we can hope given how pickle works for NumPy by https://github.com/pydata/xarray/pull/1128.
So I'm closing this now, but feel free to open another issue for any follow-up concerns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-260773846,https://api.github.com/repos/pydata/xarray/issues/1058,260773846,MDEyOklzc3VlQ29tbWVudDI2MDc3Mzg0Ng==,6213168,2016-11-15T21:26:52Z,2016-11-15T21:26:52Z,MEMBER,"Confirmed that #1017 fixes my specific issue, thanks!
Leaving the ticket open as other people (particularly those that work on large arrays without dask) will still be affected.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-256144009,https://api.github.com/repos/pydata/xarray/issues/1058,256144009,MDEyOklzc3VlQ29tbWVudDI1NjE0NDAwOQ==,1217238,2016-10-25T19:05:01Z,2016-10-25T19:05:01Z,MEMBER,"I answered the StackOverflow question:
https://stackoverflow.com/questions/13746601/preserving-numpy-view-when-pickling/40247761#40247761
This was a tricky puzzle to figure out!
","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-256123953,https://api.github.com/repos/pydata/xarray/issues/1058,256123953,MDEyOklzc3VlQ29tbWVudDI1NjEyMzk1Mw==,5635139,2016-10-25T18:23:35Z,2016-10-25T18:23:35Z,MEMBER,"@crusaderky right, I see. All those views are in the same pickle object, and so shouldn't be duplicated. That is frustrating.
As per @shoyer, the easiest way is to just not have the data in the first place. So not needing indexes at all should solve your case.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-255952251,https://api.github.com/repos/pydata/xarray/issues/1058,255952251,MDEyOklzc3VlQ29tbWVudDI1NTk1MjI1MQ==,6213168,2016-10-25T06:54:02Z,2016-10-25T06:54:02Z,MEMBER,"@maximilianr, if you pickle 2 plain python objects A and B together, and one of the attributes of B is a reference to A, A does not get duplicated.
In this case there must be some specific __getstate__ code to prevent this and/or something with the C implementation of the class
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-255801010,https://api.github.com/repos/pydata/xarray/issues/1058,255801010,MDEyOklzc3VlQ29tbWVudDI1NTgwMTAxMA==,5635139,2016-10-24T17:03:08Z,2016-10-24T17:03:08Z,MEMBER,"If I'm understanding you correctly @crusaderky, I think this is a tough problem, and one much broader than xarray. When pickling something with a reference, do you want to save the object, or the reference? If you pickle the reference, how can you guarantee to have the object available when unpickling? How would you codify the reference (memory location?)?
Is that right? Or am I misunderstanding your problem?
On this narrow case, I think not having indexes at all should solve this, though
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754
https://github.com/pydata/xarray/issues/1058#issuecomment-255622303,https://api.github.com/repos/pydata/xarray/issues/1058,255622303,MDEyOklzc3VlQ29tbWVudDI1NTYyMjMwMw==,1217238,2016-10-23T23:27:09Z,2016-10-23T23:27:09Z,MEMBER,"The plan is stop making default indexes with `np.arange`. See https://github.com/pydata/xarray/pull/1017, which is my top priority for the next major release.
I'm not confident that your work around will work properly. At the very least, you should check `strides` as well. Otherwise `get_base(array[::-1])` would return `array`.
If it would really help, I'm open to making `Variable(dims, array)` reuse the same numpy array instead of creating a view (see [`as_compatible_data`](https://github.com/pydata/xarray/blob/f0c7203e314e855ce8878ce516d618ba07772a11/xarray/core/variable.py#L147)).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,184722754