AWS SDK for node.js not picking up credentials when running in ECS

Question

I have this code:

const uploadAllToS3 = () => { console.log("upload useless file to s3, just to test access rights"); new AWS.S3().putObject({ Bucket: scheduledJobsConstants.s3BucketName(), Key: `test/${new Date()}.txt`, Body: "Hello!", }, (err) => { console.log(`env:${inspect(process.env)} provider:${AWS.CredentialProviderChain.defaultProviders} cred:${inspect(AWS.config.credentials)} err:${inspect(err)}`.replace(/(\r\n|\n|\r)/gm, "")); }); };

When this code runs in ecs(which assumes a role that has access to the S3 bucket), I will get this in the log:

The env part of the log:

env: { PATH: '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', HOSTNAME: '485bbd95f87d', AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: '/v2/credentials/be3d4d7c-28b6-47e6-8081-cd746d95cb28', ECS_CONTAINER_METADATA_FILE: '/opt/ecs/metadata/8c0b731d-49ce-4e19-bf79-64087b433876/ecs-container-metadata.json', NODE_VERSION: '8.9.4', YARN_VERSION: '1.3.2', NPM_CONFIG_PROGRESS: 'false', HOME: '/root' }

The "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" is present in env var, so I assume the sdk should be able to pick that up.

The provider part of the log is just 4 functions that's defined in the sdk code, the 4th of which would return "ECSCredentials" if it sees "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" in env var.

Strangely the cred part of the log is "null" in ECS. When I run this locally(where I manually assume a role), the cred part of the log is "EnvironmentCredentials", and the file gets uploaded to S3.

The err part of the log is:

{ Forbidden: null at Request.extractError (/app/node_modules/aws-sdk/lib/services/s3.js:557:35) at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:105:20) at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:77:10) at Request.emit (/app/node_modules/aws-sdk/lib/request.js:683:14) at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12) at /app/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:685:12) at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:115:18) message: null, code: 'Forbidden', region: null, time: 2018-01-14T06:30:00.490Z, requestId: '4E7121D88BA0FEF1', extendedRequestId: 'D5uiXf5EI/OLHeaPhYX67C384ba3SF1I950N78hJWw8Vv4XGQ0opSLOQlXaxVFW31g252dx8YUc=', cfId: undefined, statusCode: 403, retryable: false, retryDelay: 118.50320949834776 }

Is there anything wrong with my code or my env var?

Should I print more logs to help diagnose this problem?

Please advise, thanks.

==========Update==========

I have ran this code in ECS const curlCommand = `curl http://169.254.170.2${process.env.AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}`; exec(curlCommand, (error, stdout, stderr) => { console.log(`${curlCommand} err:${error} out:${stdout} stderr:${stderr}`); });

The result I got is: { "SecretAccessKey": "long string", "Token": "very long string", "Expiration": "2018-01-14T08:55:04+0000", "AccessKeyId": "shorted string" }

So if the sdk had called http://169.254.170.2${process.env.AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}, it should have gotten a correct response.

Are you sure that the provider being used is the same on your local and on the ec2 instances ? For example the ec2 instance might have got an invalid cred from the metadata service — Malice
– Malice, Commented Jan 14, 2018 at 8:08
@Malice AWS.config.credentials is null when running in ECS. my assumption is that this should not be null since "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" is present in env var. is that a correct assumption? — Cui Pengfei 崔鹏飞
– Cui Pengfei 崔鹏飞, Commented Jan 14, 2018 at 8:12
You might need to initializa AWS.EC2MetadataCredentials and assign it to aws.config.credentials. As far as I know, it doesnt do this unless it is done explicitly — Malice
– Malice, Commented Jan 14, 2018 at 8:27
@Malice thank you for the doc link. my code is not running in EC2, it's running in ECS containers. according to this: docs.aws.amazon.com/AWSJavaScriptSDK/latest/… it should load ecs credentials if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI exists, which is the case judging by my log — Cui Pengfei 崔鹏飞
– Cui Pengfei 崔鹏飞, Commented Jan 14, 2018 at 8:32

Malice · Accepted Answer · 2018-01-14 09:07:06Z

2

The simple answer is to do

AWS.config.credentials = new AWS.ECSCredentials(options);

presumably because credentials for aws-sdk credentials were not configured/ configured incorrectly.

answered Jan 14, 2018 at 9:07

Malice

1,48218 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Cui Pengfei 崔鹏飞 Over a year ago

Problem solved. It was somehow related to node version. My code was running in ECS, with a base docker image: node:8.9.4. I switched to node:9.4, which is the latest version, that solved the problem. Don't know why, but it's working now.

Aleksi · Accepted Answer · 2020-08-11 10:47:39Z

Try using AWS.ECSCredentials.get or getPromise to make sure credentials are loaded before you use them:

const ecsCredentials = new AWS.ECSCredentials({ ... }); await ecsCredentials.getPromise(); AWS.config.credentials = ecsCredentials;

Check this thread for more info: https://github.com/aws/aws-sdk-js/issues/3281#issuecomment-637171993.

redgeoff · Accepted Answer · 2022-11-29 22:56:24Z

I am using CloudFormation and in my case, I had to specify TaskRoleArn in the TaskDefinition.Properties before the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI was injected into my container. Once it was, the AWS SDK automatically picked up my credentials.

Collectives™ on Stack Overflow

AWS SDK for node.js not picking up credentials when running in ECS

3 Answers 3

1 Comment

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Related