issues: 337267315

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
337267315 MDExOlB1bGxSZXF1ZXN0MTk4NTA5NTk0 2261 xarray.backends refactor 1217238 closed 0     20 2018-07-01T06:20:27Z 2018-10-11T03:07:54Z 2018-10-09T04:13:42Z MEMBER   0 pydata/xarray/pulls/2261

A major refactor of xarray backend classes:

  • A new least-recently-used cache facilitates keeping track of open files rather than requiring them to be reopened.
  • A number of internal simplifications/clean-ups that should make backends easier to write and maintain:
    • PickleByReconstructionWrapper and DataStorePickleMixin have been eliminated in favor of CachingFIleManager.
    • Backends no longer store any state beyond their wrapped files. This fixes a bug where dask graphs were accidentally being serialized in pickled backend classes (oops!).
    • Locks are now setup by backend classes directly, rather than separately in to_netcdf/open_dataset.

xref #2121 fixes https://github.com/pydata/xarray/issues/1738 fixes https://github.com/pydata/xarray/issues/2376

  • [x] Tests added
  • [x] Tests passed

Benchmark numbers: before after ratio [66a8f8dd] [c8fbadcc] + 65.27ms 89.91ms 1.38 dataset_io.IOReadSingleNetCDF3.time_load_dataset_netcdf4 + 866.65ms 960.37ms 1.11 dataset_io.IOReadMultipleNetCDF3Dask.time_load_dataset_netcdf4_with_time_chunks_multiprocessing - 676.18ms 610.92ms 0.90 dataset_io.IOReadMultipleNetCDF3Dask.time_load_dataset_netcdf4_with_block_chunks - 61.16ms 55.20ms 0.90 dataset_io.IOReadSingleNetCDF4.time_load_dataset_netcdf4 - 167.54ms 150.41ms 0.90 dataset_io.IOReadSingleNetCDF3Dask.time_load_dataset_netcdf4_with_time_chunks - 1.11s 981.78ms 0.89 dataset_io.IOReadMultipleNetCDF3Dask.time_load_dataset_scipy_with_time_chunks - 247.82ms 219.50ms 0.89 dataset_io.IOReadSingleNetCDF3Dask.time_load_dataset_netcdf4_with_block_chunks_vindexing - 81.04ms 71.70ms 0.88 dataset_io.IOReadMultipleNetCDF4Dask.time_open_dataset_netcdf4_with_block_chunks_multiprocessing - 79.17ms 70.01ms 0.88 dataset_io.IOReadMultipleNetCDF4Dask.time_open_dataset_netcdf4_with_time_chunks_multiprocessing - 224.41ms 198.10ms 0.88 dataset_io.IOWriteMultipleNetCDF3.time_write_dataset_scipy - 67.25ms 59.26ms 0.88 dataset_io.IOReadMultipleNetCDF3.time_open_dataset_netcdf4 - 462.51ms 402.37ms 0.87 dataset_io.IOReadSingleNetCDF3Dask.time_load_dataset_scipy_with_block_chunks - 80.73ms 68.47ms 0.85 dataset_io.IOReadMultipleNetCDF4Dask.time_open_dataset_netcdf4_with_time_chunks - 221.97ms 184.93ms 0.83 dataset_io.IOReadSingleNetCDF4Dask.time_load_dataset_netcdf4_with_block_chunks_vindexing - 1.00s 825.75ms 0.82 dataset_io.IOWriteSingleNetCDF3.time_write_dataset_netcdf4 - 462.15ms 376.22ms 0.81 dataset_io.IOReadSingleNetCDF3Dask.time_load_dataset_scipy_with_time_chunks - 3.27s 2.63s 0.81 dataset_io.IOReadSingleNetCDF3Dask.time_load_dataset_scipy_with_block_chunks_oindexing - 125.08ms 99.78ms 0.80 dataset_io.IOReadSingleNetCDF3.time_load_dataset_scipy - 105.12ms 82.64ms 0.79 dataset_io.IOReadMultipleNetCDF3Dask.time_open_dataset_scipy_with_block_chunks - 112.96ms 88.28ms 0.78 dataset_io.IOReadMultipleNetCDF3.time_open_dataset_scipy - 25.36ms 19.09ms 0.75 dataset_io.IOReadSingleNetCDF3.time_orthogonal_indexing - 204.93ms 142.00ms 0.69 dataset_io.IOWriteSingleNetCDF3.time_write_dataset_scipy - 26.73ms 17.96ms 0.67 dataset_io.IOReadSingleNetCDF3.time_vectorized_indexing - 102.62ms 68.54ms 0.67 dataset_io.IOReadMultipleNetCDF3Dask.time_open_dataset_scipy_with_time_chunks - 212.34ms 139.11ms 0.66 dataset_io.IOReadSingleNetCDF3Dask.time_load_dataset_scipy_with_block_chunks_vindexing - 100.13ms 62.25ms 0.62 dataset_io.IOReadMultipleNetCDF3Dask.time_open_dataset_netcdf4_with_time_chunks - 822.44ms 467.54ms 0.57 dataset_io.IOReadMultipleNetCDF3.time_load_dataset_scipy - 666.34ms 274.17ms 0.41 dataset_io.IOReadMultipleNetCDF3.time_load_dataset_netcdf4 - 593.33ms 214.17ms 0.36 dataset_io.IOWriteNetCDFDask.time_write - 4.15s 1.49s 0.36 dataset_io.IOWriteNetCDFDaskDistributed.time_write

 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2261/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     13221727 pull

Links from other tables