Skip to main content
Ben Nadel at InVision In Real Life (IRL) 2018 (Hollywood, CA) with: Sara Dunnack
Ben Nadel at InVision In Real Life (IRL) 2018 (Hollywood, CA) with: Sara Dunnack ( @SunnE_D )

Uploading Files To Amazon S3 Using Pre-Signed (Query String Authentication) URLs

By on
Tags:

In the past, I've looked at generating pre-signed (Query String Authentication) URLs as a means to grant download-access to files on Amazon S3 (Simple Storage Service). But, pre-signed URLs can also be used to grant upload-access to files on Amazon S3. This is similar to an S3 upload policy; but it's far easier to generate due to its less flexible nature.

To demonstrate, I'm going to generate a pre-signed URL for a specific S3 resource. Then, I'm going to use ColdFusion's HTTP client to upload (ie, "PUT") a binary value to said resource. When we generate the signature for this pre-signed request, take note that I am defining the Content-Type of the upload. Unlike a GET-based pre-signed URL, which cannot send a Content-Type (and therefore cannot include it in the signature), a PUT-based request does have the opportunity to define the Content-Type of the upload.

<cfscript>

	/**
	* I get the expiration in seconds based on the given expires-at date. This takes
	* care of the UTC conversion and expects to receive a date in local time.
	*
	* @output false
	*/
	public numeric function getExpirationInSeconds( required date expiresAt ) {

		var localEpoch = dateConvert( "utc2local", "1970/01/01" );

		return( dateDiff( "s", localEpoch, expiresAt ) );

	}


	/**
	* I generate the signature for the given resource which will be available until
	* the given expiration date (in seconds).
	*
	* For GET requests, the contentType is expected to be the empty-string; for PUT
	* requests, the contentType is expected to match one of the HTTP request headers.
	*
	* @output false
	*/
	public string function generateSignature(
		required string method,
		required string contentType,
		required string resource,
		required numeric expirationInSeconds
		) {

		var stringToSignParts = [
			ucase( method ),
			"",
			contentType,
			expirationInSeconds,
			resource
		];

		var stringToSign = arrayToList( stringToSignParts, chr( 10 ) );

		var signature = hmac( stringToSign, aws.secretKey, "HmacSHA1", "utf-8" );

		// By default, ColdFusion returns the Hmac in Hex; we need to convert it to
		// base64 for usag in the pre-signed URL.
		return(
			binaryEncode( binaryDecode( signature, "hex" ), "base64" )
		);

	}


	/**
	* I encode the given S3 object key for use in a url. Amazon S3 keys have some non-
	* standard behavior for encoding - see this Amazon forum thread for more information:
	* https://forums.aws.amazon.com/thread.jspa?threadID=55746
	*
	* @output false
	*/
	public string function urlEncodeS3Key( required string key ) {

		key = urlEncodedFormat( key, "utf-8" );

		// At this point, we have a key that has been encoded too aggressively by
		// ColdFusion. Now, we have to go through and un-escape the characters that
		// AWS does not expect to be encoded.

		// The following are "unreserved" characters in the RFC 3986 spec for Uniform
		// Resource Identifiers (URIs) - http://tools.ietf.org/html/rfc3986#section-2.3
		key = replace( key, "%2E", ".", "all" );
		key = replace( key, "%2D", "-", "all" );
		key = replace( key, "%5F", "_", "all" );
		key = replace( key, "%7E", "~", "all" );

		// Technically, the "/" characters can be encoded and will work. However, if the
		// bucket name is included in this key, then it will break (since it will bleed
		// into the S3 domain: "s3.amazonaws.com%2fbucket"). As such, I like to unescape
		// the slashes to make the function more flexible. Plus, I think we can all agree
		// that regular slashes make the URLs look nicer.
		key = replace( key, "%2F", "/", "all" );

		// This one isn't necessary; but, I think it makes for a more attactive URL.
		// --
		// NOTE: That said, it looks like Amazon S3 may always interpret a "+" as a
		// space, which may not be the way other servers work. As such, we are leaving
		// the "+"" literal as the encoded hex value, %2B.
		key = replace( key, "%20", "+", "all" );

		return( key );

	}


	// ------------------------------------------------------ //
	// ------------------------------------------------------ //


	// Include my AWS credentials (so they are not in the code). Creates the structure:
	// * aws.bucket
	// * aws.accessID
	// * aws.secretKey
	include "./credentials.cfm";

	// Define the upload location (key) of the file.
	key = urlEncodeS3Key( "signed-urls/upload-test/monkey.jpg" );

	// Define the full resource of our key in our bucket.
	resource = ( "/" & aws.bucket & "/" & key );

	// Define the expiration after which this pre-signed URL is no longer valid (and will
	// be rejected by AWS).
	expirationInSeconds = getExpirationInSeconds( dateAdd( "n", 30, now() ) );

	// Generate the signature for the query-string authentication.
	// --
	// NOTE: The content-type in the signature has to match the content type in the
	// outgoing HTTP request headers.
	signature = generateSignature( "PUT", "image/jpg", resource, expirationInSeconds );

	urlEncodedSignature = urlEncodedFormat( signature );

	// Create our pre-signed URL from the various parts.
	preSignedUrl = "https://s3.amazonaws.com#resource#?AWSAccessKeyId=#aws.accessID#&Expires=#expirationInSeconds#&Signature=#urlEncodedSignature#";


	// ------------------------------------------------------ //
	// ------------------------------------------------------ //


	// Now that we have our pre-signed URL, we can use it to upload a file to Amazon S3.
	// Note that the pre-signed URL is specific to a given file and expiration date. As
	// such, this doesn't grant free-range; but, rather very targeted access based on
	// both the resource key and the file type.
	uploadRequest = new Http(
		method = "put",
		url = preSignedUrl,
		getAsBinary = "yes"
	);

	// This header is required if the contentType is non-empty in the signature.
	uploadRequest.addParam(
		type = "header",
		name = "Content-Type",
		value = "image/jpg"
	);

	uploadRequest.addParam(
		type = "body",
		value = fileReadBinary( expandPath( "./monkey.jpg" ) )
	);

	result = uploadRequest.send();

	// Output the results of the pre-signed URL upload.
	writeOutput( result.getPrefix().statusCode );
	writeOutput( "<br />" );
	writeOutput( charsetEncode( result.getPrefix().fileContent, "utf-8" ) );

</cfscript>

As you can see, the URL of the HTTP PUT is the pre-signed URL. And, when we run the above code, we get the following CFDump output:

200 OK

The image binary was successfully upload to Amazon S3 and resides at the resource defined in the pre-signed URL. By default, the uploaded object is "private"; if you want to make it public, you have to add additional Amazon Headers to the request (though, I have not tried this personally).

While the Amazon S3 upload policy allows for a lot more flexibility, uploading files using a pre-signed URL is certainly much more straightforward. Now that I see this, I'll have to go back and try to refactor some of my "upload policy" experiments.

Want to use code from this post? Check out the license.

Reader Comments

15,663 Comments

@All,

So, last night, I tried to play around with HTTP Form posts to Amazon S3 using pre-signed URLs; but, it looks like that isn't supported. In some older documentation for POST, I found this:

> Query string authentication is not supported for POST.

And, it seems rather complicated to get PUT to work from the browser.

That said, it seems like a good use-case for the PUT-based upload with pre-signed URLs is if you need to pass an upload URL to some other server-side process that needs to upload a file but doesn't necessarily have access to your credentials (such as a Message Queue job).

1 Comments

Very helpful example!
I was relying on <cffile> operations and was experiencing an issue where .jpg files were uploaded with a metadata content type of application/octet.
This example contained the content type definition I was looking for.

I believe in love. I believe in compassion. I believe in human rights. I believe that we can afford to give more of these gifts to the world around us because it costs us nothing to be decent and kind and understanding. And, I want you to know that when you land on this site, you are accepted for who you are, no matter how you identify, what truths you live, or whatever kind of goofy shit makes you feel alive! Rock on with your bad self!
Ben Nadel