home / github / issue_comments

Menu
  • GraphQL API
  • Search all tables

issue_comments: 627882905

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/4043#issuecomment-627882905 https://api.github.com/repos/pydata/xarray/issues/4043 627882905 MDEyOklzc3VlQ29tbWVudDYyNzg4MjkwNQ== 48764870 2020-05-13T10:01:08Z 2020-05-13T10:01:08Z NONE

I followed your recommendations @rabernat, please see my test code bellow.

```python import xarray as xr import os from datetime import datetime, timedelta import pandas as pd import shutil import numpy as np import time

lonlat_box = [-4.5, -2.5, 44, 45]

ERA5 IHdata - Local

-------------------

ds = xr.open_mfdataset(['raw/Wind_ERA5_Global_1998.05.nc', 'raw/Wind_ERA5_Global_1998.06.nc']) ds = ds.get('u')

from 0º,360º to -180º,180º

ds['lon'] = (ds.lon + 180) % 360 - 180

lat is upside down --> sort ascending

ds = ds.sortby(['lon', 'lat'])

Make the selection

ds = ds.sel(lon=slice(lonlat_box[0], lonlat_box[1]), lat=slice(lonlat_box[2], lonlat_box[3]))

print(ds)

tic = time.perf_counter() df = ds.to_dataframe() toc = time.perf_counter() print(f"\nLocal Network - Elapsed time: {(toc - tic)/60:0.4f} minutes\n\n")

del ds, df

ERA5 IHdata - Opendap

---------------------

ds = xr.open_mfdataset(['http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.05.nc', 'http://193.144.213.180:8080/thredds/dodsC/Wind/Wind_ERA5/Global/Wind_ERA5_Global_1998.06.nc'], chunks={'time': '500MB'})

ds = ds.get('u')

from 0º,360º to -180º,180º

ds['lon'] = (ds.lon + 180) % 360 - 180

lat is upside down --> sort ascending

ds = ds.sortby(['lon', 'lat'])

Make the selection

ds = ds.sel(lon=slice(lonlat_box[0], lonlat_box[1]), lat=slice(lonlat_box[2], lonlat_box[3]))

print(ds)

tic = time.perf_counter() df = ds.to_dataframe() toc = time.perf_counter() print(f"\n OpenDAP - Elapsed time: {(toc - tic)/60:0.4f} minutes\n\n")

del ds, df Result:ipython <xarray.DataArray 'u' (lat: 5, lon: 9, time: 1464)> dask.array<getitem, shape=(5, 9, 1464), dtype=float32, chunksize=(5, 9, 744), chunktype=numpy.ndarray> Coordinates: * lon (lon) float32 -4.5 -4.25 -4.0 -3.75 -3.5 -3.25 -3.0 -2.75 -2.5 * lat (lat) float32 44.0 44.25 44.5 44.75 45.0 * time (time) datetime64[ns] 1998-05-01 ... 1998-06-30T23:00:00 Attributes: units: m s**-1 long_name: 10 metre U wind component

Local Network - Elapsed time: 0.4037 minutes

<xarray.DataArray 'u' (lat: 5, lon: 9, time: 1464)> dask.array<getitem, shape=(5, 9, 1464), dtype=float32, chunksize=(5, 9, 120), chunktype=numpy.ndarray> Coordinates: * lon (lon) float32 -4.5 -4.25 -4.0 -3.75 -3.5 -3.25 -3.0 -2.75 -2.5 * lat (lat) float32 44.0 44.25 44.5 44.75 45.0 * time (time) datetime64[ns] 1998-05-01 ... 1998-06-30T23:00:00 Attributes: units: m s**-1 long_name: 10 metre U wind component

OpenDAP - Elapsed time: 8.1971 minutes

```

Using this chunk of time=500Mb the code runs properly but it is really slow compared with the response through local network. I will try to raise this limit in the Opendap configuration with our IT-team to a more reasonable limit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  614144170
Powered by Datasette · Queries took 0.644ms · About: xarray-datasette