Is your feature request related to a problem?

Hi, I'm trying to use DataArray.curvefit to fit a bunch of data. Let's say the data dimensions are (x, experiment_index), and I'm trying to fit m * x + b, where m will be different for each experiment_index. I would like to supply an initial guess p0 to curvefit that depends on experiment_index, but it seems like this is not supported. Here's a minimal example:

```python import numpy as np import xarray as xr

x = xr.DataArray(coords=[("x", np.linspace(0, 10, 101))]).x i = xr.DataArray(coords=[("experiment_index", [1, 2, 3])]).experiment_index

data = 2.0 * i * x + 5

m_guess = 2 * i

data.curvefit( "x", lambda x, m, b: m * x + b, p0={"m": m_guess} # I would like to provide a guess for 'm' as a function of experiment_index ) ```

Describe the solution you'd like

I would like to be able to provide arrays as the values of p0, so that I can have different initial guesses for different slices of the data.

I suppose this could also be implemented for bounds.

Describe alternatives you've considered

I could wrap curvefit in a for-loop, for example

python result = [] for y in data.transpose("experiment_index", ...): result.append(y.curvefit( "x", lambda x, m, b: m * x + b, p0={"m": m_guess.sel(experiment_index=y.experiment_index).item()}, )) result = xr.concat(result, dim="experiment_index")

But this is quite cumbersome, especially for multidimensional data.

Additional context

The above example gives the error *** ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part. because curve_fit tries to do np.atleast_1d([m_guess, 1]), but it should be np.atleast_1d([m_guess[0], 1]).

~~The above example gives the error~~ ValueError: operands could not be broadcast together with shapes (3,) (101,) ~~which comes from scipy.curve_fit tying to compute m * x, where m is the DataArray m_guess, but x is a plain Numpy array, basically x.data.~~ - this applies for scipy 1.7.

This toy example of course works with just a scalar guess like p0={"m": 2}, but in my case the function is more complicated and fit might fail if the initial guess is too far off.

The initial guess is inserted into kwargs passed to curve_fit here: https://github.com/pydata/xarray/blob/c75ac8b7ab33be95f74d9e6f10b8173c68828751/xarray/core/dataset.py#L8659

What happened?

If I have a DataArray da which I split into a Dataset using da.to_dataset(dim), and one of the values of da[dim] also happens to be one of the dimensions of da, that variable is silently missing from the resulting dataset.

What did you expect to happen?

If a variable cannot be created because it is already a dimension, it should raise an exception, or possibly issue a warning and rename the variable, so that no data is lost.

Minimal Complete Verifiable Example

```Python import xarray as xr

da = xr.DataArray( np.zeros((3, 3)), coords={ # note how 'foo' is one of the coordinate values, and also the name of a dimension "x": ["foo", "bar", "baz"], "foo": [1, 2, 3], } )

this produces a Dataset with the variables 'bar' and 'baz', 'foo' is missing (because it is already a coordinate)

print(da.to_dataset("x"))

this produces a dataset with the variables 'foo', 'bar', and 'baz', as epected

print(da.rename({"foo": "qux"}).to_dataset("x")) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python

Output of first conversion

<xarray.Dataset> Dimensions: (foo: 3) Coordinates: * foo (foo) int64 1 2 3 Data variables: bar (foo) float64 0.0 0.0 0.0 baz (foo) float64 0.0 0.0 0.0

Output of second conversion

<xarray.Dataset> Dimensions: (qux: 3) Coordinates: * qux (qux) int64 1 2 3 Data variables: foo (qux) float64 0.0 0.0 0.0 bar (qux) float64 0.0 0.0 0.0 baz (qux) float64 0.0 0.0 0.0 ```

Anything else we need to know?

This came up when I did to_dataset("param") on the fit result returned by curvefit, and one of the data dimensions happened to be the same as one of the arguments of the function which I was fitting. I was initially very confused by this.

Environment

Let me know if you think the error message is unclear or too verbose or too fancy or something.

• [x] Closes #7823
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This is a WIP follow-up for #8079 and I think also for #7051. The pattern

python missing_dims = set(dims) - set(self.dims) if missing_dims: raise ValueError(f"Dimensions {missing_dims} not found in data dimensions {tuple(self.dims)}") occurs in many methods, with small variations in the way missing_dims is calculated, the error message, and also if it's ValueError or KeyError. So it would make sense to factor it out. But I'm not familiar enough with the context around #7051 to know how to deal with sets vs tuples, so this is just a sketch for now.

• [ ] Tests added
• [ ] User visible changes (including notable bug fixes) are documented in whats-new.rst
• [ ] New functions/methods are listed in api.rst

Hello,

I noticed that arr.min("nonexistent") raises an error with a very helpful message ValueError: 'nonexistent' not found in array dimensions ('x', 'y', 'z') while arr.idxmin("nonexistent") raises KeyError: 'Dimension "nonexistent" not in dimension' [sic]

IMO, the list of dimensions should always be shown in the error message for these kinds of errors, it makes debugging much easier. With this PR, I have implemented this behavior for all such functions that I could find.

There is quite a consistent pattern which I think could be factored out into a function, but I didn't have a clear enough picture of the structure of the whole code to do it.

I didn't fix the tests yet, I'll do it if you think this can be merged.

• [x] Searched list of issues, couldn't find one related to this
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

What happened?

If I do a scatterplot with hue_style=x, where x is not "continuous" or "discrete", the result is the same as passing hue_style="continuous".

Probably related to #7907.

What did you expect to happen?

An invalid value should raise an exception.

Minimal Complete Verifiable Example

```Python import matplotlib.pyplot as plt import numpy as np import xarray as xr

x = xr.DataArray( np.random.default_rng().random((10, 3)), coords=[ ("idx", np.linspace(0, 1, 10)), ("color", [1, 2, 3]), ] )

x.plot.scatter(x="idx", hue="color", hue_style="invalid", ax=plt.figure().gca()) plt.show() ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

I also tried this on main at 3459e6fa, the behavior is the same.

What happened?

I was trying to do a scatterplot of my data with one dimension determining the color. The dimension has only a few values so I used hue_style="discrete" to have a different color for each value. However, the resulting scatterplot has a continuous colorbar, which is the same as when I pass hue_style="continuous":

What did you expect to happen?

The colorbar should have discrete colors. I was also expecting the colors to be from the default matplotlib color palette, C0, C1, etc, when there's less than 10 items, like this:

Although the examples in the documentation show the discrete case also using viridis.

What I was really expecting is a plot like one would get by passing add_colorbar=False, add_legend=True:

But that may be a bit too automagical.

Minimal Complete Verifiable Example

```Python import matplotlib.pyplot as plt import numpy as np import xarray as xr

x = xr.DataArray( np.random.default_rng().random((10, 3)), coords=[ ("idx", np.linspace(0, 1, 10)), ("color", [1, 2, 3]), ] ) y = x + np.random.default_rng().random(x.shape)

ds = xr.Dataset({ "x": x, "y": y, })

the output is the same regardless of hue_style="discrete" or "continuous" or just leaving it out

ds.plot.scatter(x="x", y="y", hue="color", hue_style="discrete", ax=plt.figure().gca()) ```

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

This is the code for the "expected" plot: ```python from matplotlib.colors import ListedColormap

ds.plot.scatter( x="x", y="y", hue="color", hue_style="discrete", ax=plt.figure().gca(),

# these lines added in addition to the MVCE
cmap=ListedColormap(["C0", "C1", "C2"]),
vmin=0.5, vmax=3.5,
cbar_kwargs=dict(ticks=ds.color.data),

) ```

Environment

I also tried this on main at 3459e6fa, the behavior is the same.

• [x] Closes #6317 and closes #6515
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

This is a rebased version of #6515, with the arg errors = "raise" | "ignore" added to Dataset and DataArray, and with tests. Let me know if the tests should be expanded further.

Fix flaky doctest introduced in #7821, see https://github.com/pydata/xarray/pull/7821#issuecomment-1537142237.

This uses the NUMBER option to compare the output with less decimal precision. It's not part of standard doctest but an extension from pytest: https://docs.pytest.org/en/7.1.x/how-to/doctest.html#using-doctest-options

Another option would be to use ... and the built-in +ELLIPSIS option, but IMO the current version is less confusing for someone reading the example.

• [x] Closes #7768
• [x] Tests added
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst

With this PR, it's possible to pass an initial guess to curvefit that is a DataArray, which will be broadcast to the data dimensions. This way, the initial guess can vary with the data coordinates.

I also added examples of using curvefit to the documentation, both a basic example and one with the multidimensional guess.

I have a couple of questions: - Should we change the signature to p0: dict[str, float | DataArray] | None, instead of dict[str, Any] (and same for bounds)? scipy only optimizes over scalars, so I think it would be safe to assume that the values should either be those, or arrays that can be broadcast. - The usage example of curvefit is only in the docstring for DataArray, so now the docs differ between DA and dataset. But the example uses a DataArray only, so this should be ok, right?

This PR fixes the broadcasting error when trying to plot multiple step plots, like arr.plot.step(..., hue=...) or arr.plot(..., drawstyle="steps-mid"). Previously, this raised a shape error, as mentioned in https://github.com/pydata/xarray/issues/4288#issuecomment-666485140. Some other relevant work was started (but apparently unfinished) in #4868 and #4866, this doesn't implement those.

• [x] Tests added
• [x] Fixes applied
• [x] User visible changes (including notable bug fixes) are documented in whats-new.rst