home / github

Menu
  • GraphQL API
  • Search all tables

issue_comments

Table actions
  • GraphQL API for issue_comments

19 rows where user = 21049064 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, created_at (date), updated_at (date)

issue 7

  • How do I copy my array forwards in time? 7
  • assign values from `xr.groupby_bins` to new `variable` 5
  • How to select using `.where()` on a timestamp `coordinate` for forecast data with 5 dimensions 3
  • sharing dimensions across dataarrays in a dataset 1
  • Animate DataArrays 1
  • Add quantile method to groupby object 1
  • calculating cumsums on a groupby object 1

user 1

  • tommylees112 · 19 ✖

author_association 1

  • NONE 19
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
513582194 https://github.com/pydata/xarray/issues/3141#issuecomment-513582194 https://api.github.com/repos/pydata/xarray/issues/3141 MDEyOklzc3VlQ29tbWVudDUxMzU4MjE5NA== tommylees112 21049064 2019-07-21T19:45:32Z 2019-07-21T19:45:32Z NONE

Would love to! Sorry have been away this weekend. Do i just clone the repo write the code and send in a PR in a new branch?

(first PR on a public repo!)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  calculating cumsums on a groupby object 469633509
508995157 https://github.com/pydata/xarray/issues/3004#issuecomment-508995157 https://api.github.com/repos/pydata/xarray/issues/3004 MDEyOklzc3VlQ29tbWVudDUwODk5NTE1Nw== tommylees112 21049064 2019-07-07T12:17:45Z 2019-07-07T12:17:45Z NONE

Perfect thankyou!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  assign values from `xr.groupby_bins` to new `variable` 453576041
508995115 https://github.com/pydata/xarray/issues/3053#issuecomment-508995115 https://api.github.com/repos/pydata/xarray/issues/3053 MDEyOklzc3VlQ29tbWVudDUwODk5NTExNQ== tommylees112 21049064 2019-07-07T12:17:12Z 2019-07-07T12:17:12Z NONE

Thanks closing!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How to select using `.where()` on a timestamp `coordinate` for forecast data with 5 dimensions 462010865
508544556 https://github.com/pydata/xarray/issues/3053#issuecomment-508544556 https://api.github.com/repos/pydata/xarray/issues/3053 MDEyOklzc3VlQ29tbWVudDUwODU0NDU1Ng== tommylees112 21049064 2019-07-04T17:29:25Z 2019-07-04T17:29:25Z NONE

This is the greatest thing since sliced bread thankyou @spencerkclark !!

I have been referring to this constantly for the last week :D

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How to select using `.where()` on a timestamp `coordinate` for forecast data with 5 dimensions 462010865
507630032 https://github.com/pydata/xarray/issues/3018#issuecomment-507630032 https://api.github.com/repos/pydata/xarray/issues/3018 MDEyOklzc3VlQ29tbWVudDUwNzYzMDAzMg== tommylees112 21049064 2019-07-02T11:10:33Z 2019-07-02T11:10:33Z NONE

This is an awesome addition thankyou! I updated my xarray to V:0.12.2 but I am still getting an error with dataset objects

```python a = np.ones((400)) * 10 a[3:7] = 0.2 a[10:13] = 0.2 p = np.repeat(a, 25).reshape(400, 5, 5) lat = np.arange(0, 5) lon = np.arange(0, 5) time = pd.date_range('2000-01-01', freq='M', periods=p.shape[0]) d = xr.Dataset( {'precip': (['time', 'lat', 'lon'], p)}, coords={ 'lon': lon, 'lat': lat, 'time': time } )

d.groupby('time.month').quantile(q=0.1) ```

gives the error message ```


AttributeError Traceback (most recent call last) <ipython-input-38-276cfe319286> in <module> ----> 1 d.groupby('time.month').quantile(q=0.1)

AttributeError: 'DatasetGroupBy' object has no attribute 'quantile' ```

Whereas for the DataArray it works fine. ```python d.precip.groupby('time.month').quantile(q=0.1)

Out[39]: <xarray.DataArray 'precip' (month: 12)> array([10., 10., 10., 10., 10., 10., 10., 10., 10., 10., 10., 10.]) Coordinates: * month (month) int64 1 2 3 4 5 6 7 8 9 10 11 12 ```

Is this the expected behaviour?

``` INSTALLED VERSIONS


commit: None python: 3.7.0 | packaged by conda-forge | (default, Nov 12 2018, 12:34:36) [Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.2.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2

xarray: 0.12.2 pandas: 0.24.2 numpy: 1.16.4 scipy: 1.3.0 netCDF4: 1.5.1.2 pydap: None h5netcdf: None h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 nc_time_axis: None PseudonetCDF: None rasterio: 1.0.17 cfgrib: 0.9.7 iris: None bottleneck: 1.2.1 dask: 1.2.2 distributed: 1.28.1 matplotlib: 3.1.0 cartopy: 0.17.0 seaborn: 0.9.0 numbagg: None setuptools: 41.0.1 pip: 19.1 conda: None pytest: 4.5.0 IPython: 7.1.1 sphinx: 2.0.1 ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Add quantile method to groupby object 455262061
506730153 https://github.com/pydata/xarray/issues/3053#issuecomment-506730153 https://api.github.com/repos/pydata/xarray/issues/3053 MDEyOklzc3VlQ29tbWVudDUwNjczMDE1Mw== tommylees112 21049064 2019-06-28T13:17:04Z 2019-06-28T13:17:42Z NONE

Thanks! Your assumption was correct, apologies for the mistake!

This might be asking for too much but is there any way I can keep track of the forecast_horizon or initialisation_date? The only issue I can foresee is that I won't be able to distinguish between the forecasts for the same date, even though those initialised closest to the valid_time will likely have more information in them than those initialised a long time before.

What I'm asking is can I add a new dimension to my original ds so that I have the OPTION to select by either valid_time/time or by forecast_horizon/initialisation_date

So I would be looking for something like:

<xarray.Dataset> Dimensions: (forecast_horizon: 24, initialisation_date: 12, lat: 36, lon: 45, number: 51, time: 288) Coordinates: * lon (lon) float64 33.5 33.7 33.9 34.1 34.3 ... 41.65 41.85 42.05 42.25 * lat (lat) float64 -5.175 -5.176 -5.177 -5.177 ... -5.2 -5.201 -5.202 * number (number) int64 0 1 2 3 4 5 6 7 8 9 ... 42 43 44 45 46 47 48 49 50 * initialisation_date (initialisation_date) datetime64[ns] 2018-01-31 ... 2018-12-31 * forecast_horizon (forecast_horizon) timedelta64[ns] 28 days ... 215 days * time (time) datetime64[ns] 2018-04-02 2018-04-03 ... 2018-04-30 Data variables: precip (forecast_horizon, initialisation_date, number, lat, lon, time) float64 0.2684 0.8408 ... 1.7 -0.383

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How to select using `.where()` on a timestamp `coordinate` for forecast data with 5 dimensions 462010865
500165601 https://github.com/pydata/xarray/issues/3004#issuecomment-500165601 https://api.github.com/repos/pydata/xarray/issues/3004 MDEyOklzc3VlQ29tbWVudDUwMDE2NTYwMQ== tommylees112 21049064 2019-06-08T21:28:34Z 2019-06-08T21:28:34Z NONE

The best way I have found so far is: df = rank_norm.to_dataframe() bins = pd.qcut(df['rank_norm'], 5, labels=[1, 2, 3, 4, 5]) output = bins.to_xarray().to_dataset().rename({'rank_norm':'rank_quantile'})

Which returns: <xarray.Dataset> Dimensions: (lat: 10, lon: 10, time: 70) Coordinates: * lat (lat) float64 -5.175 -5.125 -5.075 ... -4.825 -4.775 -4.725 * lon (lon) float64 33.52 33.57 33.62 33.67 ... 33.87 33.92 33.97 * time (time) datetime64[ns] 2010-02-28 2010-03-31 ... 2015-11-30 Data variables: rank_quantile (lat, lon, time) int64 2 1 1 1 2 2 1 1 1 ... 1 1 2 2 1 4 2 2

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  assign values from `xr.groupby_bins` to new `variable` 453576041
499959555 https://github.com/pydata/xarray/issues/3004#issuecomment-499959555 https://api.github.com/repos/pydata/xarray/issues/3004 MDEyOklzc3VlQ29tbWVudDQ5OTk1OTU1NQ== tommylees112 21049064 2019-06-07T16:53:55Z 2019-06-08T21:11:46Z NONE

So if I want them separated into 5 percentiles (0, 20) (20,40) (40,60) (60,80) (80,100)

bins = [-0.01, 20., 40., 60., 80., np.Inf] bin_labels = ['1', '2', '3', '4, '5']

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  assign values from `xr.groupby_bins` to new `variable` 453576041
499961306 https://github.com/pydata/xarray/issues/3004#issuecomment-499961306 https://api.github.com/repos/pydata/xarray/issues/3004 MDEyOklzc3VlQ29tbWVudDQ5OTk2MTMwNg== tommylees112 21049064 2019-06-07T16:59:12Z 2019-06-07T16:59:12Z NONE

Also how do I assign the result of the xarray.core.groupby.DatasetGroupBy and the labels to a new variable?

bin_labels = ['20', '40', '60', '80'] decile_index_gpby = rank_norm.groupby_bins('rank_norm', bins=bins, labels=bin_labels) decile_index_gpby.assign() # assign_coords()

Gives me the error message: ```


ValueError Traceback (most recent call last) <ipython-input-6-0c8328bf2f77> in <module> ----> 1 decile_index_gpby = rank_norm.groupby_bins('rank_norm', bins=bins, labels=bin_labels) 2 decile_index_gpby.assign() # assign_coords()

~/miniconda3/lib/python3.7/site-packages/xarray/core/common.py in groupby_bins(self, group, bins, right, labels, precision, include_lowest, squeeze) 529 cut_kwargs={'right': right, 'labels': labels, 530 'precision': precision, --> 531 'include_lowest': include_lowest}) 532 533 def rolling(self, dim=None, min_periods=None, center=False, **dim_kwargs):

~/miniconda3/lib/python3.7/site-packages/xarray/core/groupby.py in init(self, obj, group, squeeze, grouper, bins, cut_kwargs) 249 250 if bins is not None: --> 251 binned = pd.cut(group.values, bins, **cut_kwargs) 252 new_dim_name = group.name + '_bins' 253 group = DataArray(binned, group.coords, name=new_dim_name)

~/miniconda3/lib/python3.7/site-packages/pandas/core/reshape/tile.py in cut(x, bins, right, labels, retbins, precision, include_lowest, duplicates) 239 include_lowest=include_lowest, 240 dtype=dtype, --> 241 duplicates=duplicates) 242 243 return _postprocess_for_cut(fac, bins, retbins, x_is_series,

~/miniconda3/lib/python3.7/site-packages/pandas/core/reshape/tile.py in _bins_to_cuts(x, bins, right, labels, precision, include_lowest, dtype, duplicates) 357 else: 358 if len(labels) != len(bins) - 1: --> 359 raise ValueError('Bin labels must be one fewer than ' 360 'the number of bin edges') 361 if not is_categorical_dtype(labels):

ValueError: Bin labels must be one fewer than the number of bin edges

In [7]: bin_labels = ['20', '40', '60', '80'] ...: decile_index_gpby = rank_norm.groupby_bins('rank_norm', bins=bins, labels=bin_labels) ...: decile_index_gpby.assign() # assign_coords() ...:


IndexError Traceback (most recent call last) <ipython-input-7-a4ba78018478> in <module> 1 bin_labels = ['20', '40', '60', '80'] 2 decile_index_gpby = rank_norm.groupby_bins('rank_norm', bins=bins, labels=bin_labels) ----> 3 decile_index_gpby.assign() # assign_coords()

~/miniconda3/lib/python3.7/site-packages/xarray/core/groupby.py in assign(self, kwargs) 772 Dataset.assign 773 """ --> 774 return self.apply(lambda ds: ds.assign(kwargs)) 775 776

~/miniconda3/lib/python3.7/site-packages/xarray/core/groupby.py in apply(self, func, args, kwargs) 684 kwargs.pop('shortcut', None) # ignore shortcut if set (for now) 685 applied = (func(ds, *args, kwargs) for ds in self._iter_grouped()) --> 686 return self._combine(applied) 687 688 def _combine(self, applied):

~/miniconda3/lib/python3.7/site-packages/xarray/core/groupby.py in _combine(self, applied) 691 coord, dim, positions = self._infer_concat_args(applied_example) 692 combined = concat(applied, dim) --> 693 combined = _maybe_reorder(combined, dim, positions) 694 if coord is not None: 695 combined[coord.name] = coord

~/miniconda3/lib/python3.7/site-packages/xarray/core/groupby.py in _maybe_reorder(xarray_obj, dim, positions) 468 469 def _maybe_reorder(xarray_obj, dim, positions): --> 470 order = _inverse_permutation_indices(positions) 471 472 if order is None:

~/miniconda3/lib/python3.7/site-packages/xarray/core/groupby.py in _inverse_permutation_indices(positions) 110 positions = [np.arange(sl.start, sl.stop, sl.step) for sl in positions] 111 --> 112 indices = nputils.inverse_permutation(np.concatenate(positions)) 113 return indices 114

~/miniconda3/lib/python3.7/site-packages/xarray/core/nputils.py in inverse_permutation(indices) 58 # use intp instead of int64 because of windows :( 59 inverse_permutation = np.empty(len(indices), dtype=np.intp) ---> 60 inverse_permutation[indices] = np.arange(len(indices), dtype=np.intp) 61 return inverse_permutation 62

IndexError: index 1204 is out of bounds for axis 0 with size 1000 ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  assign values from `xr.groupby_bins` to new `variable` 453576041
499958420 https://github.com/pydata/xarray/issues/3004#issuecomment-499958420 https://api.github.com/repos/pydata/xarray/issues/3004 MDEyOklzc3VlQ29tbWVudDQ5OTk1ODQyMA== tommylees112 21049064 2019-06-07T16:50:36Z 2019-06-07T16:50:36Z NONE

Why does the number of bin labels have to be one less than the number of bins?

``` bin_labels = ['20', '40', '60', '80', '100'] decile_index_gpby = rank_norm.groupby_bins('rank_norm', bins=bins, labels=bin_labels)

Out[]: ValueError: Bin labels must be one fewer than the number of bin edges ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  assign values from `xr.groupby_bins` to new `variable` 453576041
497722295 https://github.com/pydata/xarray/issues/2030#issuecomment-497722295 https://api.github.com/repos/pydata/xarray/issues/2030 MDEyOklzc3VlQ29tbWVudDQ5NzcyMjI5NQ== tommylees112 21049064 2019-05-31T14:09:35Z 2019-05-31T14:09:35Z NONE

@philippjfr

How do you change the fps in the code that you posted above

Thanks for your help!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  Animate DataArrays 309965118
460600719 https://github.com/pydata/xarray/issues/2547#issuecomment-460600719 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQ2MDYwMDcxOQ== tommylees112 21049064 2019-02-05T11:15:49Z 2019-02-05T11:15:49Z NONE

But the original question was answered so thank you very much!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
460600500 https://github.com/pydata/xarray/issues/2547#issuecomment-460600500 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQ2MDYwMDUwMA== tommylees112 21049064 2019-02-05T11:15:01Z 2019-02-05T11:15:01Z NONE

Sorry for the silence! I got pulled away to another project. Unfortunately I wasn't able to finish completing the task in xarray but I found that the easiest way around the problem was to use a combination of two functions:

```python def change_missing_vals_to_9999f(ds, variable): """ Change the missing values from np.nan to -9999.0f""" arr = ds[variable].values

# set the values to -9999
arr[np.isnan(arr)] = -9999

# reassign the values back to the array
ds[variable] = (ds[variable].dims, arr)

return ds

def change_missing_data_values(filename): """ change the values INSIDE the .nc file to -9999.0f """ assert ( filename.split(".")[-1] == "nc" ), "This function only works with .nc files. Filename: {}".format(filename) print(" Processing {} ").format(filename)

# ONLY OPEN THE DATASET ONCE
ds = xr.open_dataset(filename)
variables = ds.data_vars.keys()

for variable in variables:
    print("** Working on variable {} **".format(variable))
    ds = change_missing_vals_to_9999f(ds, variable)

# ds.map(change_missing_vals_to_9999f, variable)

# rewrite to netcdf file
ds.to_netcdf(filename)
print("** Written variables {} to filename {} **").format(variables, filename)

return

```

and then another function using the NCO command:

def change_nc_FillValue(filename): """ use the NCO command to change the fillvalue metadata in the .nc files""" command = "ncatted -a _FillValue,,m,f,-9999.0 {}".format(filename) os.system(command) print("** _FillValue changed on {} file **".format(filename)) return

RUN HERE: ``` @click.command() @click.argument("filename", type=str) def main(filename): """ Run the two commands a) change the Values INSIDE the .nc file [python, numpy, xarray] b) change the associated METADATA for the .nc file headers [nco] """ change_missing_data_values(filename) change_nc_FillValue(filename)

print("**** PROCESS DONE FOR {} ****").format(filename)

return

```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
436688702 https://github.com/pydata/xarray/issues/2547#issuecomment-436688702 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQzNjY4ODcwMg== tommylees112 21049064 2018-11-07T16:36:09Z 2018-11-07T16:46:01Z NONE

@spencerkclark Thanks very much that is awesome!

One final Q: How do I set the fill_value not to nan but to -1 (this is what I need for the Land Surface Model)?

The current output of ncdump -h Rg_dummy.nc is:

... variables: double time(time) ; time:_FillValue = NaN ; time:standard_name = "time" ; time:units = "day as %Y%m%d.%f" ; time:calendar = "proleptic_gregorian" ; short Rg(time, y, x) ; Rg:_FillValue = NaN ; Rg:long_name = "HWSD sub sum content" ; Rg:units = "percent wt" ; Rg:valid_range = 97., 103. ; double latitude(y, x) ; latitude:_FillValue = -99999. ; double longitude(y, x) ; longitude:_FillValue = -99999. ;

What I want is:

ncdump -h Rg_dummy.nc ... variables: double time(time) ; time:_FillValue = -1s ; time:standard_name = "time" ; time:units = "day as %Y%m%d.%f" ; time:calendar = "proleptic_gregorian" ; short Rg(time, y, x) ; Rg:_FillValue = -1s ; Rg:long_name = "HWSD sub sum content" ; Rg:units = "percent wt" ; Rg:valid_range = 97., 103. ; double latitude(y, x) ; latitude:_FillValue = -99999. ; double longitude(y, x) ; longitude:_FillValue = -99999. ;

I want to do something like: ds2.to_netcdf(filename, set_fill_value=-1)

I saw these: Issue #1598 Issue #1865

But failed to understand where/how to use them

Thank you so much for helping me out with xarray. It's crazy powerful. It's also just very big!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
436668695 https://github.com/pydata/xarray/issues/2547#issuecomment-436668695 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQzNjY2ODY5NQ== tommylees112 21049064 2018-11-07T15:45:22Z 2018-11-07T15:46:15Z NONE

That all worked great until I tried to write out to a .nc file.

```python data_dir = "./" filename = "Rg_dummy.nc"

get the datetime range

times = pd.date_range("2000-01-01", "2000-12-31", name="time") var = "Rg"

copyfile(data_dir + filename, "temp.nc") ds = xr.open_dataset("temp.nc") print("Temporary Data read to Python")

FORWARD FILL FROM THE ORIGINAL DATA to new timesteps

ds['time'] = np.array([times[0]]) ds.reindex({"time":times}) ds.ffill("time")

ds.to_netcdf(filename, format="NETCDF3_CLASSIC") print(filename, "Written!")

remove temporary file

os.remove(data_dir+"temp.nc") print("Temporary Data Removed")

del ds ```

I get the following Error message: ``` Temporary Data read to Python


TypeError Traceback (most recent call last) <ipython-input-228-e3d645224353> in <module>() 15 ds.ffill("time") 16 ---> 17 ds.to_netcdf(filename, format="NETCDF3_CLASSIC") 18 print(filename, "Written!") 19

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/core/dataset.pyc in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute) 1148 engine=engine, encoding=encoding, 1149 unlimited_dims=unlimited_dims, -> 1150 compute=compute) 1151 1152 def to_zarr(self, store=None, mode='w-', synchronizer=None, group=None,

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/api.pyc in to_netcdf(dataset, path_or_file, mode, format, group, engine, writer, encoding, unlimited_dims, compute) 721 try: 722 dataset.dump_to_store(store, sync=sync, encoding=encoding, --> 723 unlimited_dims=unlimited_dims, compute=compute) 724 if path_or_file is None: 725 return target.getvalue()

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/core/dataset.pyc in dump_to_store(self, store, encoder, sync, encoding, unlimited_dims, compute) 1073 1074 store.store(variables, attrs, check_encoding, -> 1075 unlimited_dims=unlimited_dims) 1076 if sync: 1077 store.sync(compute=compute)

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/common.pyc in store(self, variables, attributes, check_encoding_set, unlimited_dims) 366 self.set_dimensions(variables, unlimited_dims=unlimited_dims) 367 self.set_variables(variables, check_encoding_set, --> 368 unlimited_dims=unlimited_dims) 369 370 def set_attributes(self, attributes):

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in set_variables(self, args, kwargs) 405 def set_variables(self, args, kwargs): 406 with self.ensure_open(autoclose=False): --> 407 super(NetCDF4DataStore, self).set_variables(*args, kwargs) 408 409 def encode_variable(self, variable):

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/common.pyc in set_variables(self, variables, check_encoding_set, unlimited_dims) 403 check = vn in check_encoding_set 404 target, source = self.prepare_variable( --> 405 name, v, check, unlimited_dims=unlimited_dims) 406 407 self.writer.add(source, target)

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in prepare_variable(self, name, variable, check_encoding, unlimited_dims) 451 least_significant_digit=encoding.get( 452 'least_significant_digit'), --> 453 fill_value=fill_value) 454 _disable_auto_decode_variable(nc4_var) 455

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.createVariable()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.init()

TypeError: illegal primitive data type, must be one of ['i8', 'f4', 'f8', 'S1', 'i2', 'i4', 'u8', 'u4', 'u1', 'u2', 'i1'], got datetime64[ns] ```

and if I try with the default netcdf writing options

python ds.to_netcdf(filename) print(filename, "Written!")

I get this error message: ``` Temporary Data read to Python


ValueError Traceback (most recent call last) <ipython-input-229-453d5f074d33> in <module>() 15 ds.ffill("time") 16 ---> 17 ds.to_netcdf(filename) 18 print(filename, "Written!") 19

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/core/dataset.pyc in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute) 1148 engine=engine, encoding=encoding, 1149 unlimited_dims=unlimited_dims, -> 1150 compute=compute) 1151 1152 def to_zarr(self, store=None, mode='w-', synchronizer=None, group=None,

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/api.pyc in to_netcdf(dataset, path_or_file, mode, format, group, engine, writer, encoding, unlimited_dims, compute) 721 try: 722 dataset.dump_to_store(store, sync=sync, encoding=encoding, --> 723 unlimited_dims=unlimited_dims, compute=compute) 724 if path_or_file is None: 725 return target.getvalue()

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/core/dataset.pyc in dump_to_store(self, store, encoder, sync, encoding, unlimited_dims, compute) 1073 1074 store.store(variables, attrs, check_encoding, -> 1075 unlimited_dims=unlimited_dims) 1076 if sync: 1077 store.sync(compute=compute)

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/common.pyc in store(self, variables, attributes, check_encoding_set, unlimited_dims) 366 self.set_dimensions(variables, unlimited_dims=unlimited_dims) 367 self.set_variables(variables, check_encoding_set, --> 368 unlimited_dims=unlimited_dims) 369 370 def set_attributes(self, attributes):

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in set_variables(self, args, kwargs) 405 def set_variables(self, args, kwargs): 406 with self.ensure_open(autoclose=False): --> 407 super(NetCDF4DataStore, self).set_variables(*args, kwargs) 408 409 def encode_variable(self, variable):

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/common.pyc in set_variables(self, variables, check_encoding_set, unlimited_dims) 403 check = vn in check_encoding_set 404 target, source = self.prepare_variable( --> 405 name, v, check, unlimited_dims=unlimited_dims) 406 407 self.writer.add(source, target)

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in prepare_variable(self, name, variable, check_encoding, unlimited_dims) 418 unlimited_dims=None): 419 datatype = _get_datatype(variable, self.format, --> 420 raise_on_invalid_encoding=check_encoding) 421 attrs = variable.attrs.copy() 422

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in _get_datatype(var, nc_format, raise_on_invalid_encoding) 99 def _get_datatype(var, nc_format='NETCDF4', raise_on_invalid_encoding=False): 100 if nc_format == 'NETCDF4': --> 101 datatype = _nc4_dtype(var) 102 else: 103 if 'dtype' in var.encoding:

/home/mpim/m300690/miniconda3/envs/holaps/lib/python2.7/site-packages/xarray/backends/netCDF4_.pyc in _nc4_dtype(var) 122 else: 123 raise ValueError('unsupported dtype for netCDF4 variable: {}' --> 124 .format(var.dtype)) 125 return dtype 126

ValueError: unsupported dtype for netCDF4 variable: datetime64[ns] ```

Version information

If this is useful:

``` In [230]: xr.show_versions()

INSTALLED VERSIONS

commit: None python: 2.7.15.final.0 python-bits: 64 OS: Linux OS-release: 2.6.32-696.18.7.el6.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

xarray: 0.10.7 pandas: 0.23.0 numpy: 1.11.3 scipy: 1.1.0 netCDF4: 1.4.0 h5netcdf: None h5py: None Nio: None zarr: None bottleneck: 1.2.1 cyordereddict: None dask: None distributed: None matplotlib: 1.5.1 cartopy: None seaborn: None setuptools: 39.1.0 pip: 18.1 conda: None pytest: None IPython: 5.7.0 sphinx: None ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
436606451 https://github.com/pydata/xarray/issues/2547#issuecomment-436606451 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQzNjYwNjQ1MQ== tommylees112 21049064 2018-11-07T12:24:36Z 2018-11-07T12:25:02Z NONE

The ds.time[0] won't let me set it's value to a datetime. Instead it returns a float: python In [19]: ds.time[0].item() Out[19]: 10509.0

And none of the following work: ```python

doesn't change the time value

ds.time[0].values = times[0]

returns an error because I can't assign to a function call

ds.time[0].item() = times[0]

returns ValueError: replacement data must match the Variable's shape

ds['time'].values = np.array(times[0]) ```

Thanks for your help!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
436400765 https://github.com/pydata/xarray/issues/2547#issuecomment-436400765 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQzNjQwMDc2NQ== tommylees112 21049064 2018-11-06T20:40:20Z 2018-11-06T20:41:25Z NONE

Data: netcdf_files.zip

Code below:

```python import numpy as np import pandas as pd import xarray as xr from shutil import copyfile import os

data_dir = "./" filename = "Rg_dummy.nc"

get the datetime range

times = pd.date_range("2000-01-01", "2000-12-31", name="time") var = "Rg"

copyfile(filename, "temp.nc") ds = xr.open_dataset("temp.nc") print("Temporary Data read to Python")

FORWARD FILL FROM THE ORIGINAL DATA to new timesteps

ds.reindex({"time":times}) ds.ffill("time")

ds.to_netcdf(filename, format="NETCDF3_CLASSIC")

print(filename, "Written!")

remove temporary file

os.remove(data_dir+"temp.nc") print("Temporary Data Removed")

del ds

```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
436383962 https://github.com/pydata/xarray/issues/2547#issuecomment-436383962 https://api.github.com/repos/pydata/xarray/issues/2547 MDEyOklzc3VlQ29tbWVudDQzNjM4Mzk2Mg== tommylees112 21049064 2018-11-06T19:45:46Z 2018-11-06T19:50:00Z NONE

Thanks for your help! you definitely understood me correctly!

This doesn't seem to work as it fills myRg arrays with nan values.

python ds = ds.reindex({"time":pd.date_range("2000-01-01","2000-12-31")}) ds = ds.ffill("time")

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  How do I copy my array forwards in time? 377947810
433952128 https://github.com/pydata/xarray/issues/1471#issuecomment-433952128 https://api.github.com/repos/pydata/xarray/issues/1471 MDEyOklzc3VlQ29tbWVudDQzMzk1MjEyOA== tommylees112 21049064 2018-10-29T15:21:34Z 2018-10-29T15:21:34Z NONE

@smartass101 & @shoyer what would be the code for working with a pandas.MultiIndex object in this use case? Could you show how it would work related to your example above:

<xarray.Dataset> Dimensions: (num: 21, ar:2) # <-- note that MB is still of dims {'num': 19} only Coordinates: # <-- mostly unions as done by concat * num (num) int64 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 B <U1 'r' * ar <U1 'A' 'B' # <-- this is now a dim of the dataset, but not of MA or MB Data variables: MA (num) float64 0.5 1.0 1.5 2.0 2.5 3.0 ... 8.0 8.5 9.0 9.5 10.0 10.5 MB (num) float64 1.0 1.5 2.0 2.5 3.0 3.5 ... 7.5 8.0 8.5 9.0 9.5 10.0

I am working with land surface model outputs. I have lots of one-dimensional data for different lat/lon points, at different times. I want to join them all into one dataset to make plotting easier. E.g. plot the evapotranspiration estimates for all the stations at their x,y coordinates.

Thanks very much!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  sharing dimensions across dataarrays in a dataset 241290234

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 570.449ms · About: xarray-datasette