@dcherian Is this behavior (filling with fill_value -> inserting Nans) because they share common dimensionality in terms of name, but have different coordinate values? My expectation was something that operated more like numpy broadcasting (repeating values, not filling with anything else).

I can understand how xarray's data model yields this behavior, but in that case it might be good to improve the docs for xarray.broadcast, because it says nothing about the behavior that (seem to me) mimics xarray.align.

@ethanrd @haileyajohnson @tdrwenski any thoughts on the above question?

So I'll say that taking my dataset and running nc.squeeze() was illustrative to the problem here. I see now that if a Dataset has scalar coordinates, they apply to every DataArray/Variable inside. So I see what prompts the current behavior.

I think the mismatch in my mental model is due to me coming from netCDF CF land, where coordinates for a variable are based on: 1. Other variables that match relevant shared dimension names 2. Those explicitly listed in the coordinates attribute

I see now that xarray does NOT implement that model.

This was provoked by challenges creating a new DataArray, but I see those can be solved by passing in both dims and coords (when coords contains some scalar coords).

IMO, it's not correctly implementing the rule as you phrased it. You said "still present", which isn't the case here since the coordinate wasn't present before.

The behavior I'd advocate for is that a subsetting/selection operation should never add new coordinates that weren't previously present. That by itself would be less surprising. It would also help make things more sensible given that the coordinate is only added currently in the scalar case--if you ask for more data, the coordinate isn't added, which is also unexpected given the scalar case.

I'm also not sure why _FillValue and missing_value should be required to have the same value.

We elected not to start rebuilding things with netCDF 4.9.0 since 4.9.1 should be out realSoonNow™️ , so I don't think there's a netcdf4 package in conda-forge that has it yet.

That just worked fine for me. What version of libnetcdf and netcdf4 do you have installed?

@DanCodigaMWRA Well, given that it's failing with ncdump, you can skip netcdf4 and go straight to netcdf-c.

I just created a new Python 3.7 environment on my Mac and that worked fine. What do these show? ``` ❯ conda list curl

packages in environment at /Users/rmay/miniconda3/envs/py37:

Name Version Build Channel

curl 7.83.1 h23f1065_0 conda-forge libcurl 7.83.1 h23f1065_0 conda-forge ❯ conda list certifi

packages in environment at /Users/rmay/miniconda3/envs/py37:

Name Version Build Channel

ca-certificates 2022.6.15 h033912b_0 conda-forge certifi 2022.6.15 py37hf985489_0 conda-forge Though I agree this is not an xarray problem at this point. Does it work if you do (with the env active): ncdump -h http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc ``` ?

Cc @WardF @DennisHeimbigner @haileyajohnson

$0.02 from the peanut gallery is that my mental model of Dataset is that it's a dictionary on steroids, so following the Mapping protocol (including __bool__) makes sense to me. I put "I don't know" for @shoyer's poll, but if the question was "should" then I would have said "number of variables".

While I'm not going to sit here and argue that if ds: is a common operation and important feature (I don't often open files that result in empty datasets), I'd personally want a truly compelling argument for breaking from the Mapping protocol. Right now it's "Dataset works like dict plus some stuff". It would change to be "Dataset works like dict for all things except __bool__, plus some stuff". The latter to me requires more mental effort since I'm no longer strictly supplementing past experience.

GitHub's new automated release notes may be of interest to some of this discussion. Essentially they allow you to provide a template to format the list of merged PRs on the branch since the last release.

Is this with the netcdf or with the pydap engine? If you're not sure, can you post the full error traceback?

Can you post the full traceback you get?

@saveriogzz I'm confused why you posted results for 3.6 and 3.8, given that the original issue looks like it was posted for 3.7. 🤨 At any rate, looks like your original issue, the output from show_versions() lists libnetcdf=4.7.4. That version should be fixed with regards to the _FillValue type mismatch error.

Your Python 3.8 environment does have an old version of libnetcdf. Can you try doing conda install -n py38 -c conda-forge "libnetcdf>=4.7.4" and see if that fixes your problem?

@saveriogzz what is the output of conda list libnetcdf?

You cannot use selectors with noarch conda packages. Full stop.

For conda-forge, it's perfectly fine to just unconditionally depend on the importlib_metadata backport, especially with the implementation done like here where you use the std lib version and fall back.

The metadata for that package has been patched in conda-forge/conda-forge-repodata-patches-feedstock#161 to depend on python >=3.7, so this shouldn't be an issue any more. @kim-barker How are you installing xarray/setting up your environment? What conda commands are you running? What OS/platform architecture?

On conda it's usually only done to avoid problematic/heavy weight dependencies (i.e. avoiding pyqt dependency with matplotlib-base). I'm not sure it's worth doing for pooch.

@andersy005 I'm curious, why do you go with multiple jobs within the workflow, and using artifacts to transfer state between them, rather than multiple steps in a single job?

We can avoid this by using the pypi github action thing to automatically build and upload when tagging a release on github. It uses a repo-level secret. Just a thought.

$0.02 from an outsider is that this has served us exceedingly well on MetPy. Our release process has become: 1. Close milestone 2. Adjust the auto-generated draft GitHub release (summary notes) 3. Click "publish release" -> packages uploaded to PyPI 4. Merge conda-forge update from their bots

It's almost more secure this way because the token from PyPI only has upload permissions--no need to store someone's password.

KeyError: 'tmp2m%2Etmp2m'

This looks like an issue with the encoding of the URL and what the server expects.

The message:

"The identifier `tmax.tmax%5b0%5d%5b0:3:620%5d%5b0:3:1404%5d' is not in the dataset."

makes me wonder if there's some issue with how the webserver is handling the escaping of certain characters (like e.g. [)

@alexamici Force-pushing doesn't normally close it, so this is weird...

I'd say in the case of use_ctime=True that it's a bug that it ever uses pandas for date parsing.

Probably shouldn't raise an error for 1-1-1 since that's valid according to the Climate and Forecasting netCDF conventions (see examples 4.5 and 4.6). In fact, it works perfectly when using use_cftime=True both for the original data and reading in the data from disk.

Duplicate of #1672?

Likely duplicate of #4283 .

Packaging for sphinx-autosummary-accessors looks good to me. I think the build failure is for sphinx 3.2 warnings and not the extension.

So to be clear, conda can use the --file argument to read those files and treat them as if they were dependency requirements passed on the command line. Not quite the same as an environment.yml, but I haven't had any problems so far. It's been really nice since dependabot doesn't understand the environment files.

So you can point dependabot to a directory, where for pypi it looks for files ending in .txt. You can make a subset by using a directory, or on MetPy we've remove the .txt from files we don't want dependabot to update. It feels like a bit of a hack, but it works.

So on MetPy we moved to treating our CI system as an application and pinning every direct dependency in a requirements.txt (which can be used by conda as well). We then let dependabot handle the updates. This let's us manage the updates on a package-by-package basis, where we have a single PR that lets us see what the ramifications are with regards to tests, CI, even linting.

We've been running for a limited time, but so far it has done a good job of insulating general development (coming in on PRs) from changes in the environment, which now shouldn't change on CI from run to run (yeah, yeah 2nd-order dependencies, just pin problematic ones too). For instance, for the pandas 1.1.0 breakage, we just haven't merged the PR that moves the pin there, and that has kept our doc builds green on MetPy.

cough Solving the "setuptools won't work in setup_requires because it's too late" was basically the entire driving force of pyproject.toml.

Wow...two years for pip to gain support...I would not have expected that.

I'm not here to argue, but pyproject.toml was introduced in PEP-518, which was accepted over 4 years ago. I know packaging moves slowly but I'm curious how long something has to be around before becoming "established" and ceases to be "novel". 😉

Rolling window seems fine to me. I will say that I don't generally bother bumping that on other projects until we run into an issue/new feature that necessitates it, though.

The exception looks like the identical problem as #4283 .

Looks like (to my eye anyway) it stems from: ```python import numpy as np import pandas as pd t = np.array(['2017-09-05T12:00:00.000000000', '2017-09-05T15:00:00.000000000'], dtype='datetime64[ns]') index = pd.DatetimeIndex(t)

index.get_loc(t[0].item()) # Fails with KeyError index.get_loc(t[0]) # Works `` Fails on 1.1.0. What I have no idea is whether the.item()` call is supposed to work.

Probably worth raising upstream with the THREDDS team. I do wonder if there's some issues with the chunking/compression of the native .nc files that's at play here.

Does/should any of this also consider #4212 (CuPy)?

@jacobtomlinson Any idea how this would play with the work that's been going on for units here; I'm specifically wondering if xarray ( pint ( cupy )) would/could work.

Ah, didn't realize bug fixes went in there too. And I thought I could get away without black, but I missed the quote style. All done.

cc @dcamron

Thanks for the info. Based on that, I lean towards attrs.

I think a better rationale, though, would be to formalize the role of encoding in xarray.

As a downstream user, I just want to be told what to do (assuming encoding is part of the public API for xarray). I'd love not to have to modify our code, but that's not essential necessarily.

So to clarify: is this about whether they should be in one spot or the other? Or is it about having grid_mapping and bounds in both?

This isn't accessing netCDF using opendap, it's directly accessing using HTTP. Did xarray or netCDF gain support for this and I failed to notice?

Yeah, it's not much shorter, but it's a much easier concept for new users to grasp, IMO.

Well it'd be nice to have something, because what we have to do right now is: python times[0].values.astype('datetime64[ms]').astype('O') which is an abomination and terrible for teaching.

IMO, it's always best to release code that you know will work rather than rely on upstream to get something into the next release in order for you not to be broken. I say that both as a downstream user of xarray and a library maintainer.

I just got it to render fine--I blame general GitHub flakiness around notebooks lately.

All good, thanks.

You don’t run into a problem when doing pip install xarray on Python 2. We have run into problems on our CI builds where we install from our own cache on S3, which has a list of wheels in index.html, which we install from using pip install -f <index_url> xarray. When we have an xarray wheel in there with the incorrect name, python 2.7 tries to install from it.

On Travis I have some deploy hooks: 1. Using Travis' built-in PyPI support, it uploads wheels and sdist, only on tags 2. Execute a custom script to commit built docs to github pages (not RTD). On master builds, this updates dev docs. On a tag, it adds a new directory for that version of the docs.

Travis Config Doc Deploy Script

The benefit to automation also makes it easier to distribute the workload to other people, helping with project sustainability. On my projects, I find it very nice that I make a new release on GitHub and packages appear on PyPI and the web docs are automatically updated to the new version.

I don't mind needing to update our accessor code. My only request is don't have a version that suddenly breaks it so that we only work on the old version or the new version. 😉

cc: @jthielen Might make some things in MetPy easier...

Yeah, pint doesn't do __array_function__ yet. It'd be great to get this is, if for only that it lowers the barrier to make sure xarray works with pint in the future.

Just to correct something here, missing_value is no longer considered deprecated in the current version of the standard: http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html

Near the bottom is a specific note about removing the deprecation.

So I ran into this working with a climate scientist the other day. The use case we had was given some model output that had data like: <xarray.DataArray 'NBP' (time: 1932, lat: 192, lon: 288)> [106831872 values with dtype=float32] Coordinates: * lat (lat) float32 -90.0 -89.057594 -88.11518 ... 89.057594 90.0 * lon (lon) float32 0.0 1.25 2.5 3.75 5.0 ... 355.0 356.25 357.5 358.75 * time (time) object 1850-02-01 00:00:00 ... 2011-01-01 00:00:00 we wanted to reshape to turn time into two dimensions ("year", "month"). This was to facilitate looking at the max and min as a function of "month", take the difference, and then average over "year". Is there a way to do this already that I'm not aware of?

Thanks for bringing this to the 🏁 @abrammer !

So I finally have time to work on this...but if someone has working code to do this instead, I'm totally fine with that going in instead.

I haven't had any time to start on this (and I'm a few more weeks out), so feel free to take a cut. I'm not sure what @shoyer or @rabernat have in mind for API.

@shoyer I agree with that wrapping order. I think I'd also be in favor of starting with an experiment to disable coercing to arrays.

@nbren12 The non-communicative multiplication is a consequence of operator dispatch in Python, and the reason why we want __array_function__ from numpy. Your first example dispatches to dask.array.__mul__, which doesn't know anything about pint and doesn't know how to compose its operations because there are no hooks--the pint array just gets coerced to a numpy array. The second goes to pint.Quantity.__mul__, which assumes it can wrap the dask.array (because it duck typing) and seems to succeed in doing so.

I'm not aware of any standard out there for JSON representation of netCDF, but I know it's been at least (briefly) discussed. @WardF, anything out there you're aware of?

Another spelling of this could be ds.to_dict(header_only=True), which I only suggest to mirror ncdump -h.

@lesserwhirls @dennisHeimbigner Is there any reason to expect a difference between the downloaded file and the opendap view on TDS?

Oh, I didn't even catch that the original was on defaults.

That version has the fix for the issue.

The original version of libnetcdf in @rabernat 's environment definitely had the opendap timeout issue. Not sure if that's the root cause of the problem, or not, but it's suspect.

@lesserwhirls That's an interesting idea. (@rsignell-usgs That's the one.)

@rabernat What version of the conda-forge libnetcdf package is deployed wherever you're running?

Just so I'm clear on how the workflow looks: 1. Open dataset with NetCDF/OPeNDAP 2. Serialize NetCDFDataStore (pickle? netcdf file?) 3. Ship to Dask workers 4. Reconstitute NetCDFDataStore

Certainly does seem like there's something stale in what the remote workers are getting. Confused why it works for the others, though.

I can prioritize this a bit and dig in to see what I can figure out--though I'm teaching through tomorrow. May be able to dig into this while at ECMWF.

Just haven't had the cycles to bring this to the finish line.

Why would you sort the array? Aren't you taking differences of values and dividing by differences between the matching coordinates?

That sounds reasonable to me. I don't necessarily expect all of the xarray goodness to work with those files, but I do expect them to open without error.

@rabernat While I agree that they're (somewhat) confusing files, I think you're missing two things:

1. netCDF doesn't enforce naming on dimensions and variables. Full stop. The only naming netCDF will care about is any conflict with an internal reserved name (I'm not sure that those even exist for anything besides attributes.) IMO that's a good thing, but more importantly it's not the netCDF library's job to enforce any of it.

2. CF is an attribute convention. This also means that the conventions say absolutely nothing about naming of variables and dimensions.

IMO, xarray is being overly pedantic here. XArray states that it adopts the Common Data Model (CDM); netCDF-java and the CDM were the tools used to generate the failing examples above.

I see your argument, but here's my problem. In this future where things work (assuming that NEP is accepted), and I want distributed computing with dask, units, and xarray, I have: xarray wrapping a pint array wrapping a dask array. I like composition, but that level of wrapping...feels wrong to me for some reason. Is there some elegance I'm missing here? (Other than array-like things playing together.)

And then I still need hooks in xarray so that when pint does a calculation, it can update the metadata in xarray; so it feels like we're back here anyway.

@shoyer I know elsewhere you said you weren't sure about this idea any more, but personally I'd like to push forward on this idea. Do you have problems with this approach we need to resolve? Any chance you have some preliminary code?

I think this is the right way to solve the unit issue in XArray, since at it's core unit handling is mostly a metadata operation.

Here's a sample CDL for a file: ``` netcdf temp { dimensions: profile = 1 ; station = 1 ; isobaric = 31 ; station_name_strlen = 10 ; station_description_strlen = 33 ; variables: float isobaric(station, profile, isobaric) ; isobaric:standard_name = "isobaric" ; isobaric:long_name = "isobaric" ; isobaric:units = "Pa" ; isobaric:positive = "down" ; isobaric:axis = "Z" ; float Geopotential_height_isobaric(station, profile, isobaric) ; Geopotential_height_isobaric:standard_name = "Geopotential_height_isobaric" ; Geopotential_height_isobaric:long_name = "Geopotential_height_isobaric" ; Geopotential_height_isobaric:units = "gpm" ; Geopotential_height_isobaric:coordinates = "time longitude latitude isobaric" ; char station_name(station, station_name_strlen) ; station_name:long_name = "station name" ; station_name:cf_role = "timeseries_id" ; char station_description(station, station_description_strlen) ; station_description:long_name = "station description" ; station_description:standard_name = "platform_name" ; double latitude(station) ; latitude:units = "degrees_north" ; latitude:long_name = "profile latitude" ; double longitude(station) ; longitude:units = "degrees_east" ; longitude:long_name = "profile longitude" ; double time(station, profile) ; time:units = "Hour since 2018-08-15T12:00:00Z" ; time:calendar = "proleptic_gregorian" ; time:standard_name = "time" ; time:long_name = "GRIB forecast or observation time" ;

// global attributes: :Conventions = "CDM-Extended-CF" ; :history = "Written by CFPointWriter" ; :title = "Extract Points data from Grid file /data/ldm/pub/native/grid/NCEP/GFS/Global_0p5deg/GFS_Global_0p5deg_20180815_1200.grib2.ncx3#LatLon_361X720-p25S-180p0E" ; :featureType = "timeSeriesProfile" ; :time_coverage_start = "2018-08-15T18:00:00Z" ; :time_coverage_end = "2018-08-15T18:00:00Z" ; :geospatial_lat_min = 39.9995 ; :geospatial_lat_max = 40.0005 ; :geospatial_lon_min = -105.0005 ; :geospatial_lon_max = -104.9995 ; } which gives:pytb

MissingDimensionsError Traceback (most recent call last) <ipython-input-10-d6f8d8651b9f> in <module>() 4 query.add_lonlat().accept('netcdf4') 5 nc = ncss.get_data(query) ----> 6 xr.open_dataset(NetCDF4DataStore(nc))

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs) 352 store = backends.ScipyDataStore(filename_or_obj) 353 --> 354 return maybe_decode_store(store) 355 356

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/backends/api.py in maybe_decode_store(store, lock) 256 store, mask_and_scale=mask_and_scale, decode_times=decode_times, 257 concat_characters=concat_characters, decode_coords=decode_coords, --> 258 drop_variables=drop_variables) 259 260 _protect_dataset_variables_inplace(ds, cache)

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables) 428 vars, attrs, concat_characters, mask_and_scale, decode_times, 429 decode_coords, drop_variables=drop_variables) --> 430 ds = Dataset(vars, attrs=attrs) 431 ds = ds.set_coords(coord_names.union(extra_coords).intersection(vars)) 432 ds._file_obj = file_obj

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat) 363 coords = {} 364 if data_vars is not None or coords is not None: --> 365 self._set_init_vars_and_dims(data_vars, coords, compat) 366 if attrs is not None: 367 self.attrs = attrs

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, data_vars, coords, compat) 381 382 variables, coord_names, dims = merge_data_and_coords( --> 383 data_vars, coords, compat=compat) 384 385 self._variables = variables

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/merge.py in merge_data_and_coords(data, coords, compat, join) 363 indexes = dict(extract_indexes(coords)) 364 return merge_core(objs, compat, join, explicit_coords=explicit_coords, --> 365 indexes=indexes) 366 367

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes) 433 coerced = coerce_pandas_values(objs) 434 aligned = deep_align(coerced, join=join, copy=False, indexes=indexes) --> 435 expanded = expand_variable_dicts(aligned) 436 437 coord_names, noncoord_names = determine_coords(coerced)

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/merge.py in expand_variable_dicts(list_of_variable_dicts) 209 var_dicts.append(coords) 210 --> 211 var = as_variable(var, name=name) 212 sanitized_vars[name] = var 213

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/variable.py in as_variable(obj, name) 112 'dimensions %r. xarray disallows such variables because they ' 113 'conflict with the coordinates used to label ' --> 114 'dimensions.' % (name, obj.dims)) 115 obj = obj.to_index_variable() 116

MissingDimensionsError: 'isobaric' has more than 1-dimension and the same name as one of its dimensions ('station', 'profile', 'isobaric'). xarray disallows such variables because they conflict with the coordinates used to label dimensions. ```

python import xarray as xr xr.open_dataset('http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p5deg/TwoD')

```pytb

MissingDimensionsError Traceback (most recent call last) <ipython-input-6-e2a87d803d99> in <module>() ----> 1 xr.open_dataset(gfs_cat.datasets[0].access_urls['OPENDAP'])

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs) 344 lock = _default_lock(filename_or_obj, engine) 345 with close_on_error(store): --> 346 return maybe_decode_store(store, lock) 347 else: 348 if engine is not None and engine != 'scipy':

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/backends/api.py in maybe_decode_store(store, lock) 256 store, mask_and_scale=mask_and_scale, decode_times=decode_times, 257 concat_characters=concat_characters, decode_coords=decode_coords, --> 258 drop_variables=drop_variables) 259 260 _protect_dataset_variables_inplace(ds, cache)

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables) 428 vars, attrs, concat_characters, mask_and_scale, decode_times, 429 decode_coords, drop_variables=drop_variables) --> 430 ds = Dataset(vars, attrs=attrs) 431 ds = ds.set_coords(coord_names.union(extra_coords).intersection(vars)) 432 ds._file_obj = file_obj

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/dataset.py in init(self, data_vars, coords, attrs, compat) 363 coords = {} 364 if data_vars is not None or coords is not None: --> 365 self._set_init_vars_and_dims(data_vars, coords, compat) 366 if attrs is not None: 367 self.attrs = attrs

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/dataset.py in _set_init_vars_and_dims(self, data_vars, coords, compat) 381 382 variables, coord_names, dims = merge_data_and_coords( --> 383 data_vars, coords, compat=compat) 384 385 self._variables = variables

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/merge.py in merge_data_and_coords(data, coords, compat, join) 363 indexes = dict(extract_indexes(coords)) 364 return merge_core(objs, compat, join, explicit_coords=explicit_coords, --> 365 indexes=indexes) 366 367

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/merge.py in merge_core(objs, compat, join, priority_arg, explicit_coords, indexes) 433 coerced = coerce_pandas_values(objs) 434 aligned = deep_align(coerced, join=join, copy=False, indexes=indexes) --> 435 expanded = expand_variable_dicts(aligned) 436 437 coord_names, noncoord_names = determine_coords(coerced)

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/merge.py in expand_variable_dicts(list_of_variable_dicts) 209 var_dicts.append(coords) 210 --> 211 var = as_variable(var, name=name) 212 sanitized_vars[name] = var 213

~/miniconda3/envs/py36/lib/python3.6/site-packages/xarray/core/variable.py in as_variable(obj, name) 112 'dimensions %r. xarray disallows such variables because they ' 113 'conflict with the coordinates used to label ' --> 114 'dimensions.' % (name, obj.dims)) 115 obj = obj.to_index_variable() 116

MissingDimensionsError: 'time' has more than 1-dimension and the same name as one of its dimensions ('reftime', 'time'). xarray disallows such variables because they conflict with the coordinates used to label dimensions. ```

Ah, ok, not scaling per-se (i.e. * 0.01), but a second round of value conversion.

I'm not following why the data are scaled twice.

Your point about the rounding being different is well-taken, though.

@shoyer But since it's a downstream calculation issue, and does not impact the actual precision of what's being read from the file, what's wrong with saying "Use data.astype(np.float64)". It's completely identical to doing it internally to xarray.

A float32 values has 24 bits of precision in the significand, which is more than enough to store the 16-bits in in the original data; the exponent (8 bits) will more or less take care of the * 0.01:

```python

import numpy as np np.float32(2194 * 0.01) 21.94 ```

What you're seeing is an artifact of printing out the values. I have no idea why something is printing out a float (only 7 decimal digits) out to 17 digits. Even float64 only has 16 digits (which is overkill for this application).

The difference in subtracting the 32- and 64-bit values above are in the 8th decimal place, which is beyond the actual precision of the data; what you've just demonstrated is the difference in precision between 32-bit and 64-bit values, but it had nothing to do whatsoever with the data.

If you're really worried about precision round-off for things like std. dev, you should probably calculate it using the raw integer values and scale afterwards. (I don't actually think this is necessary, though.)

My problem with custom classes (subclasses or composition) is that you will never get those from a Dataset that you get from open_dataset(). That’s a problem for me when I’m trying to make things work for users who want to do simple scripted data analysis. I’m much more interested in #1938 (or #1118).

I’m also looking at moving from pint to unyt, which is Yt’s unit support, brought into a standalone package. Beyond some performance benefits (I’ve heard), it has the benefit of being a ndarray subclass, which means asanyarray will leave it alone. That would seem to make it easier to cram into a DataArray.

Versioneer has worked great for us. Cutting a release is triggered just by making a new release on GitHub.

Since this can wait for 0.10.5, I can go slowly and get this right.

Any chance this makes it into 0.10.4?

Regarding the testing issue, another option is to use something like vcrpy to record and playback http responses for opendap requests. I've had good luck with that for Siphon.

@shoyer Happy to put in a PR to address...provided you can tell me what that should actually look like. 😁

Reworked due to corner case with PyNio on Python 2.7. Final solution looks simpler to me.

Fixed flake8 (unnecessary import)

Rebased on current master.

Test failures look unrelated--they also happen to me locally. Installing SciPy 1.0.0 locally fixes them, so I'm guessing they're caused by SciPy 1.0.1.

So did this remove/rename LazilyIndexedArray in 0.10.2? Because I'm getting an attribute in the custom xarray backend I wrote: https://github.com/Unidata/siphon/blob/master/siphon/cdmr/xarray_support.py

I don't mind updating, but I wanted to make sure this was intentional.

Somehow managed to forget about an issue I was previously involved with. Lovely. Closing in favor of that issue.

Right. But such hooks would be sufficient to properly maintain the units attribute on a DataArray and check whether math made sense. This could use pint under the covers.

At this point I'd be happy to have hooks that let me intercept/wrap ufunc operations, though I guess that's what #1938 is supporting in a more systematic way.

This looks awesome. Thoughts on where you think units fits in here?

@tlechauve One way or another, I'm sure @lesserwhirls would be interested to hear your thoughts on THREDDS.

I don't have much preference between the two--I picked one and it worked for me. I just fixed up the initial import cleanups by hand and didn't find the need for a tool to do so.

In my experience, having formatting errors hold up a PR hasn't been a problem. IMO, import ordering is a more important style issue than many of the things that flake8 catches.

I recommend flake8-import-order. If you install that plugin, then flake8 will enforce import ordering and grouping.

Given the definitions of these two, as I understand them, I think it makes sense to mask both values.