0

I have a python lambda function that is using requests to scrape metrics from Prometheus, which is http. I began by zipping the requests packages and a python function on my local machine. I uploaded and attached the zip file as a layer to the lambda function. When I test the function it times out on the GET request

import requests import copy import json print("Imported packages successfull" ) # Grabbing metadata on metrics categoriies print("Reading URL") URL_meta = 'http://xxx.xxx.x.xxx:9091/api/v1/metadata' print("URL read successfull") print("Initiating requests.get") r = requests.get(url = URL_meta) <--------- Times out print("successful requests.get") metadata = r.json()

I then made a VPC 10.0.0.0/16, a Public and Private Subnet which I attached to their respective Public and Private Route tables, an Internet Gateway attached to the VPC, a Public NAT Gateway connected to the Public Subnet, I made a Role and attached the permission policies: AWSLambdaExecute and AWSLambdaVPCAccessExecutionRole. I then went to the Lambda Function > Configuration > VPC, and added the VPC I made, attached the Private Subnet and the default VPC Security group to that. I then went to Lambda Function > Configuration > Permissions, and assigned the Execution Role.

When I run the function it times out at the same location. Here are the logs:

Function Logs START RequestId: aa1e5ffa-ed04-4274-9a66-a5ac455b5325 Version: $LATEST Imported packages successfull Reading URL URL read successfull Initiating requests.get END RequestId: xxxxxxxxx

I have successfully run the same requests function from a Jupyter Notebook on my local machine and have verified that the http address can be accessed from various devices.

I have increased the timeout period to 30 seconds. I have searched google, I have checked SO:

Cannot use Requests-Module on AWS Lambda

Issue with VPC attached to Lambda function

Why is my Lambda unable to access the internet?

How to make a HTTP rest call in AWS lambda using python?

Python request in AWS Lambda timing out

get request Task timed out in Lambda using Python

I've watched numerous YouTube videos, poured over the AWS Docs, read Medium articles, and I checked the "Similar Questions" that were suggested when I initiated this. I am stuck.

Improve this question
8
  • 1
    Let's start simple... If the Lambda function is NOT attached to a VPC, does the GET request timeout? Also, is there a particular reason why you want to attach the Lambda function to a VPC? Commented Aug 8, 2022 at 4:14
  • Yes it times out without a VPC. I only have one attached because it timed out on the GET request @JohnRotenstein Commented Aug 8, 2022 at 4:24
  • 2
    Do not attach a VPC unless you have a need to access resources inside the VPC. When a VPC is not connected, the Lambda function has direct access to the Internet. It is possible that the website you are connecting to is blocking requests coming from AWS. If you login to an EC2 instance and try to curl that URL, does it work? Commented Aug 8, 2022 at 4:43
  • 1
    It looks like you are calling an official Prometheus API, so they are unlikely to be blocking it. Commented Aug 8, 2022 at 4:54
  • 2
    I notice in the Prometheus documentation that the examples refer to curl 'http://localhost:9090/api.... This suggests that you need to install and run Prometheus on your own server. What domain name are you pointing to? Is it to your own computer? Is it accessible from the Internet? I would again suggest connecting to it from an EC2 instance (even just using curl to see if it responds). Commented Aug 8, 2022 at 6:45
Load 7 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.