issue_comments: 702635437

Thanks for the troubleshooting. I encountered the problem within an sync worker in django channels (has been and is still working with s3fs=0.4.1 ) (same error). So I tried running the refactored relevant code in a vanilla python script on a linux docker container with WSL2 backend, no event loops and no threads running. Same results and the reason I posted in in xarray issues.

No coming back to this problem a day the vanilla script works perfectly sound and gets the job down! The Django sync worker example still runs into problems:

[ERROR] asyncio 2020-10-02 09:40:10,950 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:11,609 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:12,478 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:12,743 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:12,745 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:12,757 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:13,631 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:13,633 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:15,025 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state [ERROR] asyncio 2020-10-02 09:40:16,317 :: Exception in callback BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000)) handle: <Handle BaseSelectorEventLoop._sock_connect_cb(<Future finished result=None>, <socket.socke...8.0.4', 9000)>, ('172.18.0.4', 9000))> Traceback (most recent call last): File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.7/asyncio/selector_events.py", line 510, in _sock_connect_cb fut.set_result(None) asyncio.base_futures.InvalidStateError: invalid state Exception in thread Thread-1: Traceback (most recent call last): File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/local/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.7/asyncio/base_events.py", line 541, in run_forever self._run_once() File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1771, in _run_once handle = self._ready.popleft() IndexError: pop from an empty deque

I guess this is now more of a problem with the way django >=3.1 and particularly django-channels is dealing with the event loop. Do you have by any chance quick thoughts on this? Is there a way to get the django "superpowered" event loop instead of the asyncio one?

Anyway will mark this bug report as resolved as it is cleary not xarrays issue. Thanks a lot !