issue_comments

Thanks @adanb13

@jhamman yes, I think it's alright to close. The issue seems to arise from the use of .tolist(), having to convert every value to a python float so that it may be retuned in a dictionary as is acceptable for JSON, causing the memory spike.

will try @Illviljan suggestion (thanks!) , to_json calls .compute but not sure if it calls .tolist(), in case it avoids it, that should result in greater memory efficiency.

@adanb13 - do you have plans to revisit this PR? If not, do you mind if we close it for now? Based on the comments above, I think an issue discussing the use case and potential solutions would be a good next step.

How about converting the dataset to dask dataframe? python ddf = ds.to_dask_dataframe() ddf.to_json(filename)

Use cases would be in any web service that would like to provide the final data values back to a user in JSON.

For what it's worth, I think your users will have a poor experience with encoded JSON data for very large arrays. It will be slow to compress and transfer this data.

In the long term, you would probably do better to transmit the data in some binary form (e.g., by calling tobytes() on the underlying np.ndarray objects, or by using Xarray's to_netcdf).

Thanks for report and the PR!

This really needs a "minimal complete verifiable" example (e.g., by creating and loading a Zarr array with random data) so others can verify your reported the performance gains: https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports https://stackoverflow.com/help/minimal-reproducible-example

To be honest, this fix looks a little funny to me, because NumPy's own implementation of tolist() is so similar. I would love to understand what is going on.

If you can reproduce the issue only using NumPy, it could also make more sense to file this as a upstream bug report to NumPy. The NumPy maintainers are in a better position to debug tricky memory allocation issues involving NumPy.

I'm not sure if this breaks the data model of xarray leaving inconsistent sizes?

Also this seems like a very corner usecase, I don't think it is intended to write DataArrays in ASCII.

But I let some more senior devs of xarray be the judge here :)

Made these for work (big data, government). Is useful when trying to provide data values back to end user after all data manipulation has been done. (Aka the initial Xarray.DataArray is not longer needed)

best native solution that exists (from what I see) is .to_dict() which is memory inefficient. (Had memory errors at work when trying, hence all the tests provided in the gist). Basically a user can call to_dict with data = false and then add the data values using the above 2 functions to the resulting dictionary in a more memory efficient way.

Use cases would be in any web service that would like to provide the final data values back to a user in JSON.

The failing doctest is unrelated, you can ignore it.

I'm not sure if this breaks the data model of xarray leaving inconsistent sizes?

Also this seems like a very corner usecase, I don't think it is intended to write DataArrays in ASCII.

But I let some more senior devs of xarray be the judge here :)

ran python -m pip uninstall urllib3-secure-extra for Doctests fail as suggested. Get the following message: WARNING: Skipping urllib3-secure-extra as it is not installed.