Here's my code:

```python import numpy as np import xarray as xr

da = xr.DataArray(np.random.rand(1000, 100)) da = da.rename({'dim_0': 'rows_a'}) db = xr.DataArray(np.random.rand(1000, 100)) db = db.rename({'dim_0': 'rows_b'})

def print_shape(a): print(a.shape) return np.zeros(shape=(a.shape[0]))

def print_two_shapes(a, b): print(a.shape) print(b.shape) return np.zeros(shape=(a.shape[0], b.shape[0])) ```

If I print_shape and print_shapes with apply_ufunc I am surprised by the results:

```python xr.apply_ufunc( print_shape, da, input_core_dims=[['dim_1']] )

(1000, 100) <xarray.DataArray (rows_a: 1000)> array([0., 0., 0., ..., 0., 0., 0.]) Coordinates: * rows_a (rows_a) int64 0 1 2 3 4 5 6 7 ... 992 993 994 995 996 997 998 999 ```

vs

```python xr.apply_ufunc( print_two_shapes, da, db, input_core_dims=[['dim_1'], ['dim_1']] )

(1000, 1, 100) (1000, 100)

<xarray.DataArray (rows_a: 1000, rows_b: 1000)> array([[0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], ..., [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.]]) Coordinates: * rows_a (rows_a) int64 0 1 2 3 4 5 6 7 ... 992 993 994 995 996 997 998 999 * rows_b (rows_b) int64 0 1 2 3 4 5 6 7 ... 992 993 994 995 996 997 998 999 ```

My array da has changed shape from (1000, 100) to (1000, 1, 100) when a second argument was added to my call to apply_ufunc.

Maybe this is documented, but I missed it. If it is documented I'd be glad to be pointed to it and I'll see if I can come up with a suggestion of how to highlight this better in the documentation as it really threw me.

Ran into a problem writing my xarray to zarr that @jhamman helped me figure out the source of my error.

I had set up an xarray (from a chunked dask array)

<xarray.DataArray 'rechunk-merge-ac8a0511ce784baf57cf304ebdc4a296' (snippets: 792848, symbols: 282)> dask.array<shape=(792848, 282), dtype=float64, chunksize=(5000, 282)> Coordinates: * snippets (snippets) object '0.gravatar.com||gprofiles.js||Gravatar.init' ... 'подолÑ\x8cÑ\x81к-админиÑ\x81Ñ\x82Ñ\x80аÑ\x86иÑ\x8f.Ñ\x80Ñ\x84||wp-embed.min.js||c' * symbols (symbols) object 'AnalyserNode.channelCount' ... 'window.sessionStorage'

Upon trying to write

python array.to_dataset(name='data').to_zarr('test.zarr')

I would get a memory error

```python-traceback

MemoryError Traceback (most recent call last) <ipython-input-33-ae022291811c> in <module> ----> 1 array.to_dataset(name='data').to_zarr('test.zarr')

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/core/dataset.py in to_zarr(self, store, mode, synchronizer, group, encoding, compute, consolidated) 1275 return to_zarr(self, store=store, mode=mode, synchronizer=synchronizer, 1276 group=group, encoding=encoding, compute=compute, -> 1277 consolidated=consolidated) 1278 1279 def unicode(self):

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/api.py in to_zarr(dataset, store, mode, synchronizer, group, encoding, compute, consolidated) 915 writer = ArrayWriter() 916 # TODO: figure out how to properly handle unlimited_dims --> 917 dump_to_store(dataset, zstore, writer, encoding=encoding) 918 writes = writer.sync(compute=compute) 919

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims) 790 791 store.store(variables, attrs, check_encoding, writer, --> 792 unlimited_dims=unlimited_dims) 793 794

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/zarr.py in store(self, variables, attributes, args, kwargs) 343 def store(self, variables, attributes, args, kwargs): 344 AbstractWritableDataStore.store(self, variables, attributes, --> 345 *args, kwargs) 346 347 def sync(self):

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims) 259 writer = ArrayWriter() 260 --> 261 variables, attributes = self.encode(variables, attributes) 262 263 self.set_attributes(attributes)

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/common.py in encode(self, variables, attributes) 203 """ 204 variables = OrderedDict([(k, self.encode_variable(v)) --> 205 for k, v in variables.items()]) 206 attributes = OrderedDict([(k, self.encode_attribute(v)) 207 for k, v in attributes.items()])

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/common.py in <listcomp>(.0) 203 """ 204 variables = OrderedDict([(k, self.encode_variable(v)) --> 205 for k, v in variables.items()]) 206 attributes = OrderedDict([(k, self.encode_attribute(v)) 207 for k, v in attributes.items()])

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/zarr.py in encode_variable(self, variable) 308 309 def encode_variable(self, variable): --> 310 variable = encode_zarr_variable(variable) 311 return variable 312

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/backends/zarr.py in encode_zarr_variable(var, needs_copy, name) 214 # TODO: allow toggling this explicitly via dtype in encoding. 215 coder = coding.strings.EncodedStringCoder(allows_unicode=False) --> 216 var = coder.encode(var, name=name) 217 var = coding.strings.ensure_fixed_length_bytes(var) 218

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/coding/strings.py in encode(self, variable, name) 60 safe_setitem(attrs, '_Encoding', string_encoding, name=name) 61 # TODO: figure out how to handle this in a lazy way with dask ---> 62 data = encode_string_array(data, string_encoding) 63 64 return Variable(dims, data, attrs, encoding)

~/miniconda3/envs/ovscrptd/lib/python3.6/site-packages/xarray/coding/strings.py in encode_string_array(string_array, encoding) 85 string_array = np.asarray(string_array) 86 encoded = [x.encode(encoding) for x in string_array.ravel()] ---> 87 return np.array(encoded, dtype=bytes).reshape(string_array.shape) 88 89

MemoryError: ```

My coordinates for 'snippets' are 800k long and include strings like

inassets1-internationsgmbh.netdna-ssl.com||gn1MljtM.11d8186d87588f8fe848.js||["./app-new/src/InterNations/Bundle/LayoutBundle/Resources/public/frontend/js/vendor/fingerprint2.js"]/</e.prototype.getRegularPlugins/</i<`

While the index only takes up 88MB @jhamman noted that

xarray is converting you list of string into a numpy array of bytestrings. The dimensions of this new array will be len(orig_list) x len(longest_string). So, the resulting array is going to be, potentially, much larger than the 88mb

So, this issue is a request to support my bizarre indexing.

I apparently have a bad habit of prepending my s3 strings with 's3://' (see #2740)

To reproduce:

```python import numpy as np import xarray as xr import s3fs s3 = s3fs.S3FileSystem(**options) store = s3fs.S3Map(root='bucket/store.zarr', s3=s3)

Create and write

a = xr.DataArray(np.random.randn(2, 3)) a.to_dataset(name='data').to_zarr(store)

Read back in

a = xr.open_zarr(store) print(a) Actual Result: <xarray.Dataset> Dimensions: () Data variables: empty `` Expected Result: my dataset, which I can then get the DataArray witha['data']`

If the 's3://' is not there everything works as expected.

I will add that the fact that it at least opens now means I believe we can close #2740.

xarray version 0.12.1 zarr version 2.3.1 s3fs version 0.2.1

The following code has an error in it:

```python import s3fs import xarray as xr

S3_DIR = 's3://my_bucket'

s3 = s3fs.S3FileSystem(**storage_options) store = s3fs.S3Map(root=f'{S3_DIR}/my_zarr_store', s3=s3) array = xr.open_zarr(store)['data'] ```

The presence of "s3://" at the beginning of the string causes to take a really really really long time (I don't have the time off hand but over 10 minutes) to return with a key error, that there is nothing at 'data', which is often a clue of a permissions error.

Without the "s3://" this returns quickly with my data.

This error occurred for me as I was opening other files with dask with code such as

python df = dd.read_parquet(f'{S3_DIR}/my_data.parquet', storage_options=storage_options)

I know that this is not technically an xarray issue. However, it is the xarray line that suffers the user experience as the s3fs just returns without any checking.

I was wondering whether the open_zarr function could be generous and inspect the root argument in the case of s3fs access and warn if 's3://' is detected.

I am also wondering what the interaction issue is that causes it to take so long for the permission type error to be returned.

ping @martindurant in case you have thoughts from the s3fs side.