45

I have an S3 bucket that contains database backups. I am creating a script to download the latest backup (and eventually restore it somewhere else), but I'm not sure how to go about only grabbing the most recent file from a bucket.

Is it possible to copy only the most recent file from an S3 bucket to a local directory using AWS CLI tools?

Improve this question
7
  • What are you gonna use for scripting? SDK, CLI or something else? Also how are you naming the back files in S3 Commented Jul 14, 2016 at 22:11
  • What do you mean by most recent? dont the files have different names? Commented Jul 14, 2016 at 22:25
  • Are you using s3 file versioning? Commented Jul 14, 2016 at 22:27
  • @error2007s I'm using CLI. The files are named in a few different formats, but it typically contains a name and date component. I could potentially work on something to parse the dates but I would prefer not to if I can avoid it Commented Jul 14, 2016 at 22:37
  • 3
    The accepted answer is a copy-and-paste duplicate of the accepted answer of the earlier question. See my comment below the accepted answer here. Commented Jun 22, 2019 at 15:11

3 Answers 3

Reset to default
27

And here is a bash script create based on @error2007s's answer. This script requires your aws profile and bucket name as variables, and downloads the latest object to your ~/Downloads folder:

#!/bin/sh PROFILE=your_profile BUCKET=your_bucket OBJECT="$(aws s3 ls --profile $PROFILE $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}')" aws s3 cp s3://$BUCKET/$OBJECT ~/Downloads/$OBJECT --profile $PROFILE
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Worth noting that AWS now has Cloud Console so you can do this directly from the AWS Console.
This might be more helpful by getting the more recent files (specify your date) from your bucket, then copy only those...using sort and tail perhaps to get the latest and list it (which could then be used for copying): aws s3api list-objects --bucket $BUCKET --query 'Contents[?LastModified>`2023-02-09`].{ Key: Key, LastModified: LastModified }' --output text | sort -k2 | tail -1 stackoverflow Hat tip. Hope this helps.
A better CLI method to get latest key per updated answer is: aws s3api list-objects-v2 --bucket $BUCKET --query 'sort_by(Contents, &LastModified)[-1].Key' --output=text
12 
FILE=`aws s3api list-objects-v2 --bucket "$BUCKET_NAME" --query 'reverse(sort_by(Contents[?contains(Key, \`$FILE_NAME_FILTER\`)], &LastModified))[:1].Key' --output=text`;aws s3 cp "s3://$BUCKET_NAME/$FILE" .

$BUCKET_NAME - is the bucket from which you want to download.

$FILE_NAME_FILTER - a string used as a filter for the name, which you want to match.

aws s3 cp " " - its in double-quotes because to also include files that have spaces in their names.

Improve this answer

Comments

9

The above solutions are using Bash. If one wants to do the same thing in PowerShell for downloading on Windows here is the script:

# This assumes AWS CLI exe is in your path. $s3location = "s3://bucket-name" $files = $(aws s3 ls $s3location --recursive | sort | select -last 3) $dlPath = "C:\TEMP" foreach ($s3FileInfo in $files) { $filename = $s3FileInfo.Split()[-1] $path = "${s3location}/${filename}" aws s3 cp $path $dlPath echo("Done downloading ${path} to ${dlPath}") }
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.