Hi,

I would like to open one netcdf file, changes some variables attributes, zlib compression and sometimes global attributes. I used to do it using netCDF4, and it worked.

Recently, i tried using xarray to perform the same job. The result are the same, but xarray always load entirely the file in memory, instead of write variable by variables.

Minimal example here

Creation of example file :

```python import xarray as xr

ds = xr.Dataset() obs=4835680 n = 20

basic_encoding = dict(zlib=True, shuffle=True, complevel=1)

some variables with scale factor

for i in range(3): vname = f"scale{i:02d}" ds[vname] = (["obs"], np.random.rand(obs).astype(np.float32)/1e3) ds[vname].encoding.update(basic_encoding) ds[vname].encoding.update({"dtype": np.uint16, "scale_factor": 0.0001, "add_offset": 0, "chunksizes": (1611894,)})

some variables without scale factor

for i in range(3): vname = f"float{i:02d}" ds[vname] = (["obs"], np.random.rand(obs).astype(np.float32)) ds[vname].encoding.update(basic_encoding) ds[vname].encoding.update({"chunksizes": (967136,)})

some variables with 2 dimensions which use more memory

for i in range(3): vname = f"matrix{i:02d}" ds[vname] = (["obs", "n"], np.random.rand(obs, n).astype(np.float32)*10) ds[vname].encoding.update(basic_encoding) ds[vname].encoding.update({"dtype": np.int16, "scale_factor": 0.01, "add_offset": 0, "chunksizes": (20000, 20)})

ds.to_netcdf("/tmp/test_original.nc") ```

here is my olf function to copy/rewrite my netcdf file, and the new function (i deleted useless changes in both function to keep only importants parts)

```python import netCDF4 def old_copy(f_in, f_out):

with netCDF4.Dataset(f_out, 'w') as h_out:
    with netCDF4.Dataset(f_in, 'r') as h_in:
        for dimension, size in h_in.dimensions.items():
            h_out.createDimension(dimension, len(size))

        for varname, var_in in h_in.variables.items():
            var_out = h_out.createVariable(
                varname, var_in.dtype, var_in.dimensions,
                zlib=True, complevel=2
            )
            for key in var_in.ncattrs():
                if key != '_FillValue':
                    setattr(var_out, key, getattr(var_in, key))
            var_in.set_auto_maskandscale(False)
            var_out.set_auto_maskandscale(False)
            var_out[:] = var_in[:]

        for attr in h_in.ncattrs():
            setattr(h_out, attr, getattr(h_in, attr))

def new_copy(f_in, f_out): with xr.open_dataset(f_in) as d_in: d_in.to_netcdf(f_out) ```

here i compare both function in term of memory usage,

```python import holoviews as hv from dask.diagnostics import ResourceProfiler, visualize hv.extension("bokeh")

F_IN = "/tmp/test_original.nc" F_OUT = "/tmp/test.nc"

!rm -rfv {F_OUT} with ResourceProfiler(dt=0.1) as rprof_old:

old_copy(F_IN, F_OUT)

rprof.visualize()

!rm -rfv {F_OUT} with ResourceProfiler(dt=0.1) as rprof_new:

new_copy(F_IN, F_OUT)

visualize([rprof_old, rprof_new]) ```

What happened:

xarray seems to load the entire file in memory to dump it.

What you expected to happen:

How can i tell xarray to load/dump variable by variable without loading the entire file ?

Thanks you

Environment:

Hi all,

if a xr.Dataset is chunked, i cannot do ds.time.min(), i get an error : ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]'). I don't know if it is expected ? Moreover, ds2.time.mean() works

Thanks

What happened:

raised an UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')

What you expected to happen:

compute the min & max on a chunked datetime64 xarray.DataArray

Minimal Complete Verifiable Example:

```python import xarray as xr import numpy as np

obs=200 t0 = np.datetime64("2010-01-01T00:00:00") tn = t0 + np.timedelta64(123*4, "D")

ds2 = xr.Dataset( { "time": (["obs"], np.arange(t0, tn, (tn-t0)/obs)), }, coords={ "obs": (["obs"], np.arange(obs)), }, ).chunk({"obs": 100})

ds2.time.min() ```

Anything else we need to know?:

ds2.time.mean() works, max & min raise Exception

Environment:

Hi all,

I start to play with xarray and zarr and came across something curious : I create a zarr folder and a zarr variable in uint32. When i load this dataset with xarray, it loads in float64. I don't know if it is something expected ?

```python fichier1 = "/tmp/test.zarr"

zh = zarr.open(fichier1, "w")

example = np.zeros(10, dtype=np.uint32) myvar = zh.create_dataset("myvar", shape=example.shape, dtype=example.dtype )

myvar.attrs["_ARRAY_DIMENSIONS"] = ["obs"] # <- without this, the zarr dataset will not be readable by xarray myvar[:] = example

dtype is uint32

zh.myvar.dtype python

dtype('uint32') ```

when reloading with zarr :

```python

dtype is stil uint32

zh = zarr.open(fichier1, 'r') zh.myvar.dtype ```

```python

dtype('uint32') ```

But when loading with xarray :

```python

dtype is float64

ds = xr.open_zarr(fichier1) ds.myvar.dtype ```

```python

dtype('float64') ```

Is it something expected ? Am I missing something ?

link to the notebook created : bad_dtype_zarr_xarray

Environment:

Hi,

I felt on this issue#2582 about the problem when arithmetic operation doesn't keep attribute in an DataArray.

Is this problem not merged yet ? I just installed a fresh conda env with python3.8 & xarray 0.16.2 and the problem still persist :

```python ds = xr.Dataset({"a": (("x",), np.array([1,2,3]))}) ds["a"].attrs["units"] = "m" ds.a

Out[1]: <xarray.DataArray 'a' (x: 3)> array([1, 2, 3]) Dimensions without coordinates: x Attributes: units: m ```

```python ds["b"] = ds.a * 2 ds.b

Out[2]: <xarray.DataArray 'b' (x: 3)> array([2, 4, 6]) Dimensions without coordinates: x ```