issue_comments

Thanks @scottyhq !

ok. as noted https://github.com/mapbox/rasterio/pull/1709 the aws_unsigned=True should bypass importing boto3 in upcoming rasterio 1.0.25. But I've gone ahead and added boto3 to ci requirements.yml files in case people want to add other tests reading from s3, gcs, etc. in the future

The build is already green because this is marked as an allowed failure, but it shows up in the list of failing tests anyways. It's a little confusing but that was the best way I found to make the result of these tests noticeable without counting as a build failure.

If you click on "Checks" at the top of this PR and then click on the top level "pydata.xarray" section of Azure, you see the full details on why the test is failing still: ``` self = <xarray.tests.test_backends.TestRasterio object at 0x7fb4b5cfa358>

@network
def test_rasterio_vrt_network(self):
    # Make sure loading w/ rasterio give same results as xarray
    import rasterio
    # use same url that rasterio package uses in tests
    prefix = "https://landsat-pds.s3.amazonaws.com/L8/139/045/"
    image = "LC81390452014295LGN00/LC81390452014295LGN00_B1.TIF"
    httpstif = prefix + image

  with rasterio.Env(aws_unsigned=True):

xarray/tests/test_backends.py:3797:

/usr/share/miniconda/envs/xarray-tests/lib/python3.6/site-packages/rasterio/env.py:193: in init aws_unsigned=aws_unsigned)

self = <rasterio.session.AWSSession object at 0x7fb4b4b10da0>, session = None aws_unsigned = True, aws_access_key_id = None, aws_secret_access_key = None aws_session_token = None, region_name = None, profile_name = None requester_pays = False

def __init__(
        self, session=None, aws_unsigned=False, aws_access_key_id=None,
        aws_secret_access_key=None, aws_session_token=None,
        region_name=None, profile_name=None, requester_pays=False):
    """Create a new boto3 session

    Parameters
    ----------
    session : optional
        A boto3 session object.
    aws_unsigned : bool, optional (default: False)
        If True, requests will be unsigned.
    aws_access_key_id : str, optional
        An access key id, as per boto3.
    aws_secret_access_key : str, optional
        A secret access key, as per boto3.
    aws_session_token : str, optional
        A session token, as per boto3.
    region_name : str, optional
        A region name, as per boto3.
    profile_name : str, optional
        A shared credentials profile name, as per boto3.
    requester_pays : bool, optional
        True if the requester agrees to pay transfer costs (default:
        False)
    """

  import boto3

E ModuleNotFoundError: No module named 'boto3' ```

It looks like using the aws_unsigned argument means that boto3 is required, but we don't have that installed in our CI currently. It would be straightforward to add it whenever we install rasterio -- put it in the lists in ci/requirements/

OK great. Let's at least try and get this passing and turn it green.

Do you have a better idea of why it's still broken? Is it an issue with rasterio (re your comment above)? Is it fixed on master? If not we should open an issue there. If so we could install a later version on that build

I'm in no rush to merge this (just wanted to help get rid of the red on Azure :). I think it's important to have network tests since it's increasingly common to be streaming data from s3, gcs, etc rather than reading locally. But I'm not sure of best practices for setting such tests up, so any suggestions for modifications are welcome.

@scottyhq would you prefer to merge this as-is as an improvement, or wait until the test is fixed?

It looks like the flakey test failed but that Azure counts it as passing: https://dev.azure.com/xarray/xarray/_build/results?buildId=389&view=logs

...similar to an allow_failures

but seems like the tests are now run with each build on Azure? https://dev.azure.com/xarray/xarray/_build/results?buildId=389&view=ms.vss-test-web.build-test-results-tab

the new test currently still fails - I think due to https://github.com/mapbox/rasterio/pull/1709

cc @shoyer who created the issue linked in the first post.

Yes, network test are not run on travis: https://github.com/pydata/xarray/blob/1a5b6305be4261f04b130a84f7780c9b5dafc943/conftest.py

Thanks @scottyhq !

Did we disable some tests / builds before? If so, shall we re-enable them as part of this PR? (I remember seeing something but wasn't involved directly)