Retrieve the last version of an artifact on S3

Assuming you have a CI build uploading your deployment artifacts to S3 you would have a folder structure similar to:

• my-bucket
• -project1
• -- my-war-1.war
• -- my-war-2.war
• -- my-war-3.war
• - project2
• -- my-scripts-1.tar.gz
• -- my-scripts-2.tar.gz
• -- my-scripts-3.tar.gz

You may have a scenario where you need to retrieve the latest version uploaded. This can easily be achieved with a combination of AWS CLI and jq.

Install AWS CLI and jq first

# Run as root
yum -y install python-pip
pip install --upgrade awscli

wget http://stedolan.github.io/jq/download/linux64/jq
mv jq /usr/bin/
chmod 755 /usr/bin/jq

And then use the ListObjects command to retrieve the list of artifacts matching the pattern project2/my-scripts and process it with jq to return the last element ordered by modification date:

MY_SCRIPTS=`aws s3api list-objects --bucket my-bucket \
 --prefix project2/my-scripts \
 | jq '.Contents | sort_by(.LastModified) \
 | reverse | .[0] | .Key' | tail -1 | sed 's/\"//g'`

aws s3api get-object --bucket my-bucket \
 --key ${MY_SCRIPTS} ~/my-scripts.tar.gz

Where the list-objects command returns something similar to

{
 "CommonPrefixes": [], 
 "Contents": [
 {
 "LastModified": "2014-04-30T22:26:01.000Z", 
 "ETag": "\"3a33ba576dbdaa28ba35f1d6ac16abb9\"", 
 "StorageClass": "STANDARD", 
 "Key": "my-scripts-1.tar.gz", 
 "Owner": {
 "DisplayName": "bruno.chauvet", 
 "ID": "e2a83092266..."
 }, 
 "Size": 249227
 }, 
 {
 "LastModified": "2014-05-01T04:05:13.000Z", 
 "ETag": "\"0debb07a38d26ee5673b15ad088d5454\"", 
 "StorageClass": "STANDARD", 
 "Key": "my-scripts-2.tar.gz", 
 "Owner": {
 "DisplayName": "bruno.chauvet", 
 "ID": "e2a83092266..."
 }, 
 "Size": 249227
 }
 ]
}

And the following jq command returns the Content.Key with the latest LastModified attribute.

jq '.Contents | sort_by(.LastModified) | reverse | .[0] | .Key' | tail -1 | sed 's/\"//g'