issue_comments: 627882905
This data as json
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | performed_via_github_app | issue |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/pydata/xarray/issues/4043#issuecomment-627882905 | https://api.github.com/repos/pydata/xarray/issues/4043 | 627882905 | MDEyOklzc3VlQ29tbWVudDYyNzg4MjkwNQ== | 48764870 | 2020-05-13T10:01:08Z | 2020-05-13T10:01:08Z | NONE | I followed your recommendations @rabernat, please see my test code bellow. ```python import xarray as xr import os from datetime import datetime, timedelta import pandas as pd import shutil import numpy as np import time lonlat_box = [-4.5, -2.5, 44, 45] ERA5 IHdata - Local-------------------ds = xr.open_mfdataset(['raw/Wind_ERA5_Global_1998.05.nc', 'raw/Wind_ERA5_Global_1998.06.nc']) ds = ds.get('u') from 0º,360º to -180º,180ºds['lon'] = (ds.lon + 180) % 360 - 180 lat is upside down --> sort ascendingds = ds.sortby(['lon', 'lat']) Make the selectionds = ds.sel(lon=slice(lonlat_box[0], lonlat_box[1]), lat=slice(lonlat_box[2], lonlat_box[3])) print(ds) tic = time.perf_counter() df = ds.to_dataframe() toc = time.perf_counter() print(f"\nLocal Network - Elapsed time: {(toc - tic)/60:0.4f} minutes\n\n") del ds, df ERA5 IHdata - Opendap---------------------ds = xr.open_mfdataset(['http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.05.nc', 'http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.06.nc'], chunks={'time': '500MB'}) ds = ds.get('u') from 0º,360º to -180º,180ºds['lon'] = (ds.lon + 180) % 360 - 180 lat is upside down --> sort ascendingds = ds.sortby(['lon', 'lat']) Make the selectionds = ds.sel(lon=slice(lonlat_box[0], lonlat_box[1]), lat=slice(lonlat_box[2], lonlat_box[3])) print(ds) tic = time.perf_counter() df = ds.to_dataframe() toc = time.perf_counter() print(f"\n OpenDAP - Elapsed time: {(toc - tic)/60:0.4f} minutes\n\n") del ds, df
Local Network - Elapsed time: 0.4037 minutes <xarray.DataArray 'u' (lat: 5, lon: 9, time: 1464)> dask.array<getitem, shape=(5, 9, 1464), dtype=float32, chunksize=(5, 9, 120), chunktype=numpy.ndarray> Coordinates: * lon (lon) float32 -4.5 -4.25 -4.0 -3.75 -3.5 -3.25 -3.0 -2.75 -2.5 * lat (lat) float32 44.0 44.25 44.5 44.75 45.0 * time (time) datetime64[ns] 1998-05-01 ... 1998-06-30T23:00:00 Attributes: units: m s**-1 long_name: 10 metre U wind component OpenDAP - Elapsed time: 8.1971 minutes ``` Using this chunk of time=500Mb the code runs properly but it is really slow compared with the response through local network. I will try to raise this limit in the Opendap configuration with our IT-team to a more reasonable limit. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
614144170 |