html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/3213#issuecomment-615499609,https://api.github.com/repos/pydata/xarray/issues/3213,615499609,MDEyOklzc3VlQ29tbWVudDYxNTQ5OTYwOQ==,1217238,2020-04-17T23:01:15Z,2020-04-17T23:01:15Z,MEMBER,"Wrapping `scipy.sparse` in xarray would present two challenges: 1. It only supports 2D arrays, which feels awkward for a library focused on N-dimensional data. 2. There is no existing ""duck array"" compatibility layer (i.e., `__array_function__`) that makes scipy.sparse matrices work like NumPy arrays (in fact, they actually are designed to mimic the deprecated np.matrix). (2) is the biggest challenge. I don't want to maintain that compatibility layer inside xarray, but if it existed we would be happy to try using it. pydata/sparse solves both these problems, though again indeed it only has quite limited data structures. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,479942077 https://github.com/pydata/xarray/issues/3213#issuecomment-526748987,https://api.github.com/repos/pydata/xarray/issues/3213,526748987,MDEyOklzc3VlQ29tbWVudDUyNjc0ODk4Nw==,1217238,2019-08-30T21:01:55Z,2019-08-30T21:01:55Z,MEMBER,"You will need to install NumPy 1.17 or set the env variable before importing NumPy. On Fri, Aug 30, 2019 at 1:57 PM firdaus janoos wrote: > Thanks. > > That solved that error but introduced another one. > > Specifically - this is my dataframe > [image: image] > > > and this is the error that I get with sparse=True > > [image: image] > > [image: image] > > > My numpy version is definitely about 1.16 > [image: image] > > > I also set this > os.environ[""NUMPY_EXPERIMENTAL_ARRAY_FUNCTION""]='1' > just in case > > Furthermore, I don't get this error when I don't set sparse=True ( I just > get OOM errors but that's another matter) ... > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > , > or mute the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,479942077 https://github.com/pydata/xarray/issues/3213#issuecomment-526718101,https://api.github.com/repos/pydata/xarray/issues/3213,526718101,MDEyOklzc3VlQ29tbWVudDUyNjcxODEwMQ==,1217238,2019-08-30T19:19:13Z,2019-08-30T19:19:13Z,MEMBER,"We have a new ""sparse=True"" option in xarray.Dataset.from_dataframe for exactly this use case. Pandas's to_xarray() method just calls this method, so it would make sense to forward keyword arguments, too. On Fri, Aug 30, 2019 at 11:53 AM firdaus janoos wrote: > Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically > creates a sparse dataarray - or we have a flag in to_xarray which allows > controlling for this. I have a very sparse dataframe and everytime I try to > convert it to xarray I blow out my memory. Keeping it sparse but logically > as a DataArray would be fantastic > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > , > or mute the thread > > . > ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",,479942077 https://github.com/pydata/xarray/issues/3213#issuecomment-521691465,https://api.github.com/repos/pydata/xarray/issues/3213,521691465,MDEyOklzc3VlQ29tbWVudDUyMTY5MTQ2NQ==,1217238,2019-08-15T15:50:42Z,2019-08-15T15:50:42Z,MEMBER,"Yes, it would be useful (eventually) to have lazy loading of sparse arrays from disk, like we want we currently do for dense arrays. This would indeed require knowing that the indices are sorted.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,479942077 https://github.com/pydata/xarray/issues/3213#issuecomment-521533999,https://api.github.com/repos/pydata/xarray/issues/3213,521533999,MDEyOklzc3VlQ29tbWVudDUyMTUzMzk5OQ==,1217238,2019-08-15T06:42:44Z,2019-08-15T06:42:44Z,MEMBER,"I like the indexed ragged array representation because it maps directly into sparse’s COO format. I’m sure other formats would be possible, but they would also likely be harder to implement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,479942077 https://github.com/pydata/xarray/issues/3213#issuecomment-521301555,https://api.github.com/repos/pydata/xarray/issues/3213,521301555,MDEyOklzc3VlQ29tbWVudDUyMTMwMTU1NQ==,1217238,2019-08-14T15:42:58Z,2019-08-14T15:42:58Z,MEMBER,"netCDF has a pretty low-level base spec, with conventions left to higher level docs like [CF conventions](http://cfconventions.org). Fortunately, there does seems to be a CF convention that would be a good fit for for sparse data in COO format, namely the [indexed ragged array representation]( http://cfconventions.org/cf-conventions/cf-conventions.html#_indexed_ragged_array_representation) ([example](http://cfconventions.org/cf-conventions/cf-conventions.html#_indexed_ragged_array_representation_of_profiles), note the `instance_dimension` attribute). That's probably the right thing to use for sparse arrays in xarray.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,479942077