html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/4478#issuecomment-704365715,https://api.github.com/repos/pydata/xarray/issues/4478,704365715,MDEyOklzc3VlQ29tbWVudDcwNDM2NTcxNQ==,3322897,2020-10-06T15:46:21Z,2020-10-06T15:46:21Z,NONE,"Welcome to the world of light-sheet microscopy. And this would be considered a tiny dataset..  😄 

@rabernat thanks for the tip. I was wondering what would be best chunk-size for s3 bucket storages. Will aim for that size once performance tweaking.

Thanks a lot! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,712782711
https://github.com/pydata/xarray/issues/4478#issuecomment-704351253,https://api.github.com/repos/pydata/xarray/issues/4478,704351253,MDEyOklzc3VlQ29tbWVudDcwNDM1MTI1Mw==,3322897,2020-10-06T15:27:56Z,2020-10-06T15:27:56Z,NONE,"Confirmed! Works like a charm. Went up all the way to (1024,1024,100,3,1) without any issues. Thanks for the fast fix! 👍 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,712782711
https://github.com/pydata/xarray/issues/4478#issuecomment-702815971,https://api.github.com/repos/pydata/xarray/issues/4478,702815971,MDEyOklzc3VlQ29tbWVudDcwMjgxNTk3MQ==,3322897,2020-10-02T15:58:44Z,2020-10-02T15:58:44Z,NONE,"Okay. That was too fast. Wasn't able to get the django implementation running, so I setup a testing environment with a docker composition running minio and a python3.8 based container with the libraries (repo attached). The setup runs fine for dask arrays of size (1024,1024,8,2,1) but causes the same asyncio problematic at (1024,1024,10,2,1) (tested with random arrays)

https://github.com/jhnnsrs/s3fs_bugreport


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,712782711
https://github.com/pydata/xarray/issues/4478#issuecomment-702635437,https://api.github.com/repos/pydata/xarray/issues/4478,702635437,MDEyOklzc3VlQ29tbWVudDcwMjYzNTQzNw==,3322897,2020-10-02T09:51:32Z,2020-10-02T09:51:32Z,NONE,"Thanks for the troubleshooting.  I encountered the problem within an sync worker in django channels (has been and is still working with s3fs=0.4.1 ) (same error).  So I tried running the refactored relevant code in a _vanilla_ python script on a linux docker container with WSL2 backend, no event loops and no threads running. Same results and the reason I posted in in xarray issues.

No coming back to this problem a day the vanilla script works perfectly sound and gets the job down! The Django sync worker example still runs into problems:

```
[ERROR]  asyncio 2020-10-02 09:40:10,950 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:11,609 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:12,478 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:12,743 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:12,745 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:12,757 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:13,631 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:13,633 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:15,025 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
[ERROR]  asyncio 2020-10-02 09:40:16,317 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))
handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))>
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/asyncio/events.py"", line 88, in _run
    self._context.run(self._callback, *self._args)
  File ""/usr/local/lib/python3.7/asyncio/selector_events.py"", line 510, in _sock_connect_cb
    fut.set_result(None)
asyncio.base_futures.InvalidStateError: invalid state
Exception in thread Thread-1:
Traceback (most recent call last):
  File ""/usr/local/lib/python3.7/threading.py"", line 926, in _bootstrap_inner
    self.run()
  File ""/usr/local/lib/python3.7/threading.py"", line 870, in run
    self._target(*self._args, **self._kwargs)
  File ""/usr/local/lib/python3.7/asyncio/base_events.py"", line 541, in run_forever
    self._run_once()
  File ""/usr/local/lib/python3.7/asyncio/base_events.py"", line 1771, in _run_once
    handle = self._ready.popleft()
IndexError: pop from an empty deque
```

I guess this is now more of a problem with the way django >=3.1  and particularly django-channels is dealing with the event loop. Do you have by any chance quick thoughts on this? Is there a way to get the django ""superpowered"" event loop instead of the asyncio one?

Anyway will mark this bug report as resolved as it is cleary not xarrays issue.
Thanks a lot !




","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,712782711