issue_comments

Agreed. I would avoid that kind of thing too. Maybe a stern warning for all conflicting attributes, and saying that they will be dropped from the new variable.

For units specifically, Python has a variety of unit libraries that wrap numpy arrays and can probably do some magic. Not sure if we really want to do that, though.

On Wed, Feb 26, 2014 at 4:07 PM, Stephan Hoyer notifications@github.comwrote:

Stern warnings about conflicting attributes like units may be an appropriate compromise. But if we go that way, I would advocate for making units drop upon doing any mathematical operation. We could try to update units automatically (e.g., kg * kg = kg^2), but that is tricky to always right.

Celsius, for example, is a pretty weird physical unit because of how it can take on negative values, so it actually makes a lot of sense to use mostly use Kelvin instead (for which I can sensibly apply any math operation like times or minus). That doesn't mean that I want to store all my raw data in degrees K, though...

On Wed, Feb 26, 2014 at 3:45 PM, ebrevdo notifications@github.com wrote:

err, which attributes conflict.

On Wed, Feb 26, 2014 at 3:45 PM, Eugene Brevdo ebrevdo@gmail.com wrote:

I don't think that example has your intended affect. I don't know why anyone would add something of units kelvin with those of celsius. I understand what you're saying, so maybe we should just throw a stern warning listing which units conflict and how, every single time.

On Wed, Feb 26, 2014 at 3:42 PM, Stephan Hoyer < notifications@github.com wrote:

I see your point, but I favor a more pragmatic approach by default. See my fourth bullet under "Design Goals" in the README and bullet ii under Iris in "Prior Art".

My vision here is a more powerful ndarray enhanced rather than limited by metadata. This is closer to what pandas does, which even allows for conflicting indices resulting in NaN values (a feature I would love to copy).

I think that both use cases can be covered as long as the merge/conflict logic is clearly documented and it is possible to write stricter logic for library code (which by necessity will be more verbose). If it is essential for units to agree before doing x + y, you can add assert x.attribubes.get('units') == y.attributes.get('units'). Otherwise, we will end up prohibiting operations like that when x has units of Celsius and y has units of Kelvin.

On Wed, Feb 26, 2014 at 3:23 PM, ebrevdo notifications@github.com wrote:

Also, there are plenty of other bits where you don't want conflicts. Imagine that you have variables indexed on different basemap projections. Creating exceptions to the rule seems like a bit of a rabbit hole.

On Wed, Feb 26, 2014 at 3:13 PM, Eugene Brevdo ebrevdo@gmail.com wrote:

This is an option, but these lists will break if we try to express other data formats using these conventions. For example, grib likely has other conventions. We would have to overload attribute or variable depending on what the underlying datastore is.

On Wed, Feb 26, 2014 at 3:03 PM, Stephan Hoyer < notifications@github.com wrote:

x + y could indeed check variable attributes before trying to do the merge. I don't know if it does in the current implementation.

My concern is more that metadata like "title" or "source" should not be required to match, because that metadata will almost always be conflicting. Perhaps "units", "_FIllValue", "scale_factor" and "add_offset" (if

values were not automatically masked/scaled) should be specifically blacklisted to prohibit conflicts.

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36189171> .

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36190935>

.

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36192859> .

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36193148>

.

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/25#issuecomment-36194729 .

err, which attributes conflict.

On Wed, Feb 26, 2014 at 3:45 PM, Eugene Brevdo ebrevdo@gmail.com wrote:

I don't think that example has your intended affect. I don't know why anyone would add something of units kelvin with those of celsius. I understand what you're saying, so maybe we should just throw a stern warning listing which units conflict and how, every single time.

On Wed, Feb 26, 2014 at 3:42 PM, Stephan Hoyer notifications@github.comwrote:

I see your point, but I favor a more pragmatic approach by default. See my fourth bullet under "Design Goals" in the README and bullet ii under Iris in "Prior Art".

My vision here is a more powerful ndarray enhanced rather than limited by metadata. This is closer to what pandas does, which even allows for conflicting indices resulting in NaN values (a feature I would love to copy).

I think that both use cases can be covered as long as the merge/conflict logic is clearly documented and it is possible to write stricter logic for library code (which by necessity will be more verbose). If it is essential for units to agree before doing x + y, you can add assert x.attribubes.get('units') == y.attributes.get('units'). Otherwise, we will end up prohibiting operations like that when x has units of Celsius and y has units of Kelvin.

On Wed, Feb 26, 2014 at 3:23 PM, ebrevdo notifications@github.com wrote:

Also, there are plenty of other bits where you don't want conflicts. Imagine that you have variables indexed on different basemap projections. Creating exceptions to the rule seems like a bit of a rabbit hole.

On Wed, Feb 26, 2014 at 3:13 PM, Eugene Brevdo ebrevdo@gmail.com wrote:

This is an option, but these lists will break if we try to express other data formats using these conventions. For example, grib likely has other conventions. We would have to overload attribute or variable depending on what the underlying datastore is.

On Wed, Feb 26, 2014 at 3:03 PM, Stephan Hoyer < notifications@github.com wrote:

x + y could indeed check variable attributes before trying to do the merge. I don't know if it does in the current implementation.

My concern is more that metadata like "title" or "source" should not be required to match, because that metadata will almost always be conflicting. Perhaps "units", "_FIllValue", "scale_factor" and "add_offset" (if

values were not automatically masked/scaled) should be specifically blacklisted to prohibit conflicts.

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36189171> .

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36190935>

.

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/25#issuecomment-36192859 .

I don't think that example has your intended affect. I don't know why anyone would add something of units kelvin with those of celsius. I understand what you're saying, so maybe we should just throw a stern warning listing which units conflict and how, every single time.

On Wed, Feb 26, 2014 at 3:42 PM, Stephan Hoyer notifications@github.comwrote:

I see your point, but I favor a more pragmatic approach by default. See my fourth bullet under "Design Goals" in the README and bullet ii under Iris in "Prior Art".

My vision here is a more powerful ndarray enhanced rather than limited by metadata. This is closer to what pandas does, which even allows for conflicting indices resulting in NaN values (a feature I would love to copy).

I think that both use cases can be covered as long as the merge/conflict logic is clearly documented and it is possible to write stricter logic for library code (which by necessity will be more verbose). If it is essential for units to agree before doing x + y, you can add assert x.attribubes.get('units') == y.attributes.get('units'). Otherwise, we will end up prohibiting operations like that when x has units of Celsius and y has units of Kelvin.

On Wed, Feb 26, 2014 at 3:23 PM, ebrevdo notifications@github.com wrote:

Also, there are plenty of other bits where you don't want conflicts. Imagine that you have variables indexed on different basemap projections. Creating exceptions to the rule seems like a bit of a rabbit hole.

On Wed, Feb 26, 2014 at 3:13 PM, Eugene Brevdo ebrevdo@gmail.com wrote:

This is an option, but these lists will break if we try to express other data formats using these conventions. For example, grib likely has other conventions. We would have to overload attribute or variable depending on what the underlying datastore is.

On Wed, Feb 26, 2014 at 3:03 PM, Stephan Hoyer < notifications@github.com wrote:

x + y could indeed check variable attributes before trying to do the merge. I don't know if it does in the current implementation.

My concern is more that metadata like "title" or "source" should not be required to match, because that metadata will almost always be conflicting. Perhaps "units", "_FIllValue", "scale_factor" and "add_offset" (if

values were not automatically masked/scaled) should be specifically blacklisted to prohibit conflicts.

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36189171> .

Reply to this email directly or view it on GitHub< https://github.com/akleeman/xray/issues/25#issuecomment-36190935>

.

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/25#issuecomment-36192859 .

Also, there are plenty of other bits where you don't want conflicts. Imagine that you have variables indexed on different basemap projections. Creating exceptions to the rule seems like a bit of a rabbit hole.

On Wed, Feb 26, 2014 at 3:13 PM, Eugene Brevdo ebrevdo@gmail.com wrote:

This is an option, but these lists will break if we try to express other data formats using these conventions. For example, grib likely has other conventions. We would have to overload attribute or variable depending on what the underlying datastore is.

On Wed, Feb 26, 2014 at 3:03 PM, Stephan Hoyer notifications@github.comwrote:

x + y could indeed check variable attributes before trying to do the merge. I don't know if it does in the current implementation.

My concern is more that metadata like "title" or "source" should not be required to match, because that metadata will almost always be conflicting. Perhaps "units", "_FIllValue", "scale_factor" and "add_offset" (if values were not automatically masked/scaled) should be specifically blacklisted to prohibit conflicts.

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/25#issuecomment-36189171 .

This is an option, but these lists will break if we try to express other data formats using these conventions. For example, grib likely has other conventions. We would have to overload attribute or variable depending on what the underlying datastore is.

On Wed, Feb 26, 2014 at 3:03 PM, Stephan Hoyer notifications@github.comwrote:

x + y could indeed check variable attributes before trying to do the merge. I don't know if it does in the current implementation.

My concern is more that metadata like "title" or "source" should not be required to match, because that metadata will almost always be conflicting. Perhaps "units", "_FIllValue", "scale_factor" and "add_offset" (if values were not automatically masked/scaled) should be specifically blacklisted to prohibit conflicts.

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/25#issuecomment-36189171 .

It depends on whether x+y does attribute checking before performing the merge. Again, if units don't match then maybe you shouldn't add. I always favor the strictest approach so you don't get strange surprises.

On Wed, Feb 26, 2014 at 2:47 PM, Stephan Hoyer notifications@github.comwrote:

Dataset.merge is also triggered by assigning a DatasetArray to a dataset or by doing a mathematical operation on two DatasetArrays (e.g., x + y). The later is how I encountered this issue today.

For merge itself, I would agree that we may want to default to stricter behavior, but for these other versions of merge we should default to something more flexible.

Reply to this email directly or view it on GitHubhttps://github.com/akleeman/xray/issues/25#issuecomment-36187723 .

I would default to 3, and in the exception suggest using a different merge option. Imagine merging two datasets with different _FillValue, unit, or compression attributes.