html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/5790#issuecomment-1008223776,https://api.github.com/repos/pydata/xarray/issues/5790,1008223776,IC_kwDOAMm_X848GEYg,13301940,2022-01-09T03:49:32Z,2022-01-09T03:49:32Z,MEMBER,"> @shoyer Tried Dask as you suggested, and it helped significantly! Thanks for the suggestion!

@zachglee, should we close this issue?

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,995207525
https://github.com/pydata/xarray/issues/5790#issuecomment-918735708,https://api.github.com/repos/pydata/xarray/issues/5790,918735708,IC_kwDOAMm_X842wstc,2448579,2021-09-14T02:12:17Z,2021-09-14T02:12:17Z,MEMBER,"You're using merge to concatenate along `C` That is indeed inefficient. Can you use `xr.concat([da1, da2], dim=""C"")` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,995207525
https://github.com/pydata/xarray/issues/5790#issuecomment-918697549,https://api.github.com/repos/pydata/xarray/issues/5790,918697549,IC_kwDOAMm_X842wjZN,1217238,2021-09-14T00:39:03Z,2021-09-14T00:39:03Z,MEMBER,"> I have a hunch that all arrays get aligned to the final merged coordinate space (which is much bigger), _before_ they are combined, which means at some point in the middle of the process we have a bunch of arrays in memory that have been inflated to the size of the final output array.

Yes, I'm pretty sure this is the case.

> If that's the case, it seems like it should be possible to make this operation more efficient by creating just one inflated array and adding the data from the input arrays to it in-place? Or is this an expected and unavoidable behavior with merging? (fwiw this also affects several other combination methods, presumably because they use `merge()` under the hood?)

Yes, I imagine this could work.

But on the other hand, the implementation would get more complex. For example, it's nice to be able to use `np.concatenate()` so things automatically work with other array backends like Dask.

By the way, if you haven't tried Dask already I would recommend it for this use-case. It can do streaming operations that can result in significant memory savings.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,995207525
https://github.com/pydata/xarray/issues/5790#issuecomment-918583565,https://api.github.com/repos/pydata/xarray/issues/5790,918583565,IC_kwDOAMm_X842wHkN,5635139,2021-09-13T21:15:56Z,2021-09-13T21:15:56Z,MEMBER,"I'll let others respond, but temporary memory usage of 3X sounds within expectations, albeit towards the higher end. 

If we can reduce it that would be great, but probably needs someone to work on this fairly methodically.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,995207525