Using aiobotocore to invoke lambdas hang after many requests

[marcioemiranda]

Hello,

This may be a duplicate of #738

I am using aiobotocore to invoke the lambda service asynchronously.
It works most of the time, but if I execute it like 200k times, a few requests will hang and be released only when the future is cancelled (I run it from a lambda. The future is cancelled when lambda times out)

The code that hangs look like this

session = get_session()
aio_lambda = session.create_client('lambda')
q = Queue()
producers = [ensure_future(produce(q, context)) for _ in range(num_producers)]
consumers = [ensure_future(consume(q, context)) for _ in range(num_consumers)]
await gather(*producers)
...

# this is called 1800 times in 3 minutes inside a consumer task. Around 10 calls will hang
response = await aio_lambda.invoke(
                FunctionName=environ['WORKER_FUNCTION_ARN'],
                InvocationType='Event',
                Payload=payload
                )

Python version : 3.6
All dependencies came from pip aioboto3
aiobotocore-0.12.0
aiohttp-3.6.2

Below is the stack trace thrown when the task is cancelled.
It looks like aiohttp was waiting forever. Can I set a timeout in aiobotocore that would affect the HTTP session in aiohttp? Maybe the default timeout is too high or is not set at all.

Traceback (most recent call last):
  File "/var/task/orchestrator/lambda_function.py", line 242, in invoke_worker
    Payload=payload
  File "/opt/python/aiobotocore/client.py", line 77, in _make_api_call
    operation_model, request_dict, request_context)
  File "/opt/python/aiobotocore/client.py", line 96, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/opt/python/aiobotocore/endpoint.py", line 74, in _send_request
    exception):
  File "/opt/python/aiobotocore/endpoint.py", line 176, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/opt/python/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/opt/python/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/opt/python/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/opt/python/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/opt/python/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "/opt/python/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/opt/python/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "/opt/python/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "/opt/python/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/opt/python/aiobotocore/endpoint.py", line 137, in _do_get_response
    http_response = await self._send(request)
  File "/opt/python/aiobotocore/endpoint.py", line 218, in _send
    request.method, url=url, headers=headers_, data=data, proxy=proxy)
  File "/opt/python/aiohttp/client.py", line 483, in _request
    timeout=real_timeout
  File "/opt/python/aiohttp/connector.py", line 499, in connect
    raise e
  File "/opt/python/aiohttp/connector.py", line 490, in connect
    await fut
concurrent.futures._base.CancelledError

[cmflynn]

did you end up resolving this?

[thehesiod]

may be worth trying with the latest aiohttp, they made a fix to connection pooling