html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/2799#issuecomment-553601146,https://api.github.com/repos/pydata/xarray/issues/2799,553601146,MDEyOklzc3VlQ29tbWVudDU1MzYwMTE0Ng==,5635139,2019-11-13T21:03:23Z,2019-11-13T21:03:23Z,MEMBER,"That's great that's helpful @nbren12 . Maybe we should add to docs (we don't really have a performance section at the moment, maybe we start something on performance tips?) There's some info on the differences in the Terminology that @gwgundersen wrote: https://github.com/pydata/xarray/blob/master/doc/terminology.rst#L18 Essentially: by indexing on the variable, you ignore the coordinates, and so skip a bunch of code that takes the object apart and puts it back together. A variable is much more similar to a numpy array, so you can't do `sel`, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-552714604,https://api.github.com/repos/pydata/xarray/issues/2799,552714604,MDEyOklzc3VlQ29tbWVudDU1MjcxNDYwNA==,5635139,2019-11-12T03:10:39Z,2019-11-12T03:10:39Z,MEMBER,"One note: if you're indexing into a dataarray and don't care about the coords, index into the variable. 2x numpy time, rather than 30x: ```python In [26]: da = xr.tutorial.open_dataset('air_temperature')['air'] In [27]: da Out[27]: <xarray.DataArray 'air' (time: 2920, lat: 25, lon: 53)> [3869000 values with dtype=float32] Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00 Attributes: long_name: 4xDaily Air temperature at sigma level 995 units: degK precision: 2 GRIB_id: 11 GRIB_name: TMP var_desc: Air temperature dataset: NMC Reanalysis level_desc: Surface statistic: Individual Obs parent_stat: Other actual_range: [185.16 322.1 ] In [20]: %timeit da.variable[0] 28.2 µs ± 2.29 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [21]: %timeit da[0] 459 µs ± 37.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In [22]: %timeit da.variable.values[0] 14.1 µs ± 183 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) ```","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-552646381,https://api.github.com/repos/pydata/xarray/issues/2799,552646381,MDEyOklzc3VlQ29tbWVudDU1MjY0NjM4MQ==,5635139,2019-11-11T22:29:58Z,2019-11-11T22:29:58Z,MEMBER,"TBC I think there's plenty we could do with relatively little complexity to speed up indexing operations on `DataArray`s. As an example, we could avoid the roundtrip to a temporary `Dataset`. That's a different problem from making xarray as fast as indexing a numpy array, or allowing libraries to iterate through a `DataArray` in a hot loop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-539100243,https://api.github.com/repos/pydata/xarray/issues/2799,539100243,MDEyOklzc3VlQ29tbWVudDUzOTEwMDI0Mw==,5635139,2019-10-07T16:39:54Z,2019-10-07T16:39:54Z,MEMBER,"Great analysis, thanks Do we have any idea of which of those lines are offending? I used a tool `line_profiler` a while ago, but maybe we know already (I'm guessing it's the two `_replace_with_new_dims` lines?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-469898607,https://api.github.com/repos/pydata/xarray/issues/2799,469898607,MDEyOklzc3VlQ29tbWVudDQ2OTg5ODYwNw==,5635139,2019-03-05T23:16:43Z,2019-03-05T23:16:43Z,MEMBER,"> Cython + memoryviews isn't quite the right comparison here. Right, tbc, I'm only referring to the top two lines of the pasted benchmark; i.e. once we enter python (even if only to access a numpy array) we're already losing a lot of the speed relative to the loop staying in C / Cython. So even if xarray were a python front-end to a C++ library, it still wouldn't be competitive if performance were paramount. ...unless pypy sped that up; I'd be v interested to see.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-469861382,https://api.github.com/repos/pydata/xarray/issues/2799,469861382,MDEyOklzc3VlQ29tbWVudDQ2OTg2MTM4Mg==,5635139,2019-03-05T21:19:31Z,2019-03-05T21:19:31Z,MEMBER,"To put the relative speed of numpy access into perspective, I found this insightful: https://jakevdp.github.io/blog/2012/08/08/memoryview-benchmarks/ (it's now a few years out of date, but I think the fundamentals still stand) Pasted from there: >Summary Here are the timing results we've seen above: > Python + numpy: 6510 ms Cython + numpy: 668 ms Cython + memviews (slicing): 22 ms Cython + raw pointers: 2.47 ms Cython + memviews (no slicing): 2.45 ms So if we're running an inner loop on an array, accessing it using numpy in python is an order of magnitude slower than accessing it using numpy in C (and that's an order of magnitude slower than using a slice, and that's an order of magnitude slower than using raw pointers) So - let's definitely speed xarray up (your benchmarks are excellent, thank you again, and I think you're right there are opportunities for significant increases). But where speed is paramount above all else, we shouldn't use _any_ access in python, let alone the niceties of xarray access.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-469449165,https://api.github.com/repos/pydata/xarray/issues/2799,469449165,MDEyOklzc3VlQ29tbWVudDQ2OTQ0OTE2NQ==,5635139,2019-03-04T22:33:03Z,2019-03-04T22:33:03Z,MEMBER,"You can always use xarray to process the data, and then extract the underlying array (`da.values`) for passing into something expecting an numpy array / for running fast(ish) loops (we do this frequently). ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458 https://github.com/pydata/xarray/issues/2799#issuecomment-469445483,https://api.github.com/repos/pydata/xarray/issues/2799,469445483,MDEyOklzc3VlQ29tbWVudDQ2OTQ0NTQ4Mw==,5635139,2019-03-04T22:20:58Z,2019-03-04T22:20:58Z,MEMBER,"Thanks for the benchmarks @nbren12, and for the clear explanation @shoyer While we could do some performance work on that loop, I think we're likely to see a material change by enabling the external library to access directly from the array, without a looped python call. That's consistent with the ideas @jhamman had a few days ago. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,416962458