Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
Ben Nadel at CFUNITED 2009 (Lansdowne, VA) with: Anita Neal
Ben Nadel at CFUNITED 2009 (Lansdowne, VA) with: Anita Neal

Uploading Files To Amazon S3 Using Pre-Signed (Query String Authentication) URLs

By Ben Nadel on
Tags: ColdFusion

In the past, I've looked at generating pre-signed (Query String Authentication) URLs as a means to grant download-access to files on Amazon S3 (Simple Storage Service). But, pre-signed URLs can also be used to grant upload-access to files on Amazon S3. This is similar to an S3 upload policy; but it's far easier to generate due to its less flexible nature.

To demonstrate, I'm going to generate a pre-signed URL for a specific S3 resource. Then, I'm going to use ColdFusion's HTTP client to upload (ie, "PUT") a binary value to said resource. When we generate the signature for this pre-signed request, take note that I am defining the Content-Type of the upload. Unlike a GET-based pre-signed URL, which cannot send a Content-Type (and therefore cannot include it in the signature), a PUT-based request does have the opportunity to define the Content-Type of the upload.

  • <cfscript>
  •  
  • /**
  • * I get the expiration in seconds based on the given expires-at date. This takes
  • * care of the UTC conversion and expects to receive a date in local time.
  • *
  • * @output false
  • */
  • public numeric function getExpirationInSeconds( required date expiresAt ) {
  •  
  • var localEpoch = dateConvert( "utc2local", "1970/01/01" );
  •  
  • return( dateDiff( "s", localEpoch, expiresAt ) );
  •  
  • }
  •  
  •  
  • /**
  • * I generate the signature for the given resource which will be available until
  • * the given expiration date (in seconds).
  • *
  • * For GET requests, the contentType is expected to be the empty-string; for PUT
  • * requests, the contentType is expected to match one of the HTTP request headers.
  • *
  • * @output false
  • */
  • public string function generateSignature(
  • required string method,
  • required string contentType,
  • required string resource,
  • required numeric expirationInSeconds
  • ) {
  •  
  • var stringToSignParts = [
  • ucase( method ),
  • "",
  • contentType,
  • expirationInSeconds,
  • resource
  • ];
  •  
  • var stringToSign = arrayToList( stringToSignParts, chr( 10 ) );
  •  
  • var signature = hmac( stringToSign, aws.secretKey, "HmacSHA1", "utf-8" );
  •  
  • // By default, ColdFusion returns the Hmac in Hex; we need to convert it to
  • // base64 for usag in the pre-signed URL.
  • return(
  • binaryEncode( binaryDecode( signature, "hex" ), "base64" )
  • );
  •  
  • }
  •  
  •  
  • /**
  • * I encode the given S3 object key for use in a url. Amazon S3 keys have some non-
  • * standard behavior for encoding - see this Amazon forum thread for more information:
  • * https://forums.aws.amazon.com/thread.jspa?threadID=55746
  • *
  • * @output false
  • */
  • public string function urlEncodeS3Key( required string key ) {
  •  
  • key = urlEncodedFormat( key, "utf-8" );
  •  
  • // At this point, we have a key that has been encoded too aggressively by
  • // ColdFusion. Now, we have to go through and un-escape the characters that
  • // AWS does not expect to be encoded.
  •  
  • // The following are "unreserved" characters in the RFC 3986 spec for Uniform
  • // Resource Identifiers (URIs) - http://tools.ietf.org/html/rfc3986#section-2.3
  • key = replace( key, "%2E", ".", "all" );
  • key = replace( key, "%2D", "-", "all" );
  • key = replace( key, "%5F", "_", "all" );
  • key = replace( key, "%7E", "~", "all" );
  •  
  • // Technically, the "/" characters can be encoded and will work. However, if the
  • // bucket name is included in this key, then it will break (since it will bleed
  • // into the S3 domain: "s3.amazonaws.com%2fbucket"). As such, I like to unescape
  • // the slashes to make the function more flexible. Plus, I think we can all agree
  • // that regular slashes make the URLs look nicer.
  • key = replace( key, "%2F", "/", "all" );
  •  
  • // This one isn't necessary; but, I think it makes for a more attactive URL.
  • // --
  • // NOTE: That said, it looks like Amazon S3 may always interpret a "+" as a
  • // space, which may not be the way other servers work. As such, we are leaving
  • // the "+"" literal as the encoded hex value, %2B.
  • key = replace( key, "%20", "+", "all" );
  •  
  • return( key );
  •  
  • }
  •  
  •  
  • // ------------------------------------------------------ //
  • // ------------------------------------------------------ //
  •  
  •  
  • // Include my AWS credentials (so they are not in the code). Creates the structure:
  • // * aws.bucket
  • // * aws.accessID
  • // * aws.secretKey
  • include "./credentials.cfm";
  •  
  • // Define the upload location (key) of the file.
  • key = urlEncodeS3Key( "signed-urls/upload-test/monkey.jpg" );
  •  
  • // Define the full resource of our key in our bucket.
  • resource = ( "/" & aws.bucket & "/" & key );
  •  
  • // Define the expiration after which this pre-signed URL is no longer valid (and will
  • // be rejected by AWS).
  • expirationInSeconds = getExpirationInSeconds( dateAdd( "n", 30, now() ) );
  •  
  • // Generate the signature for the query-string authentication.
  • // --
  • // NOTE: The content-type in the signature has to match the content type in the
  • // outgoing HTTP request headers.
  • signature = generateSignature( "PUT", "image/jpg", resource, expirationInSeconds );
  •  
  • urlEncodedSignature = urlEncodedFormat( signature );
  •  
  • // Create our pre-signed URL from the various parts.
  • preSignedUrl = "https://s3.amazonaws.com#resource#?AWSAccessKeyId=#aws.accessID#&Expires=#expirationInSeconds#&Signature=#urlEncodedSignature#";
  •  
  •  
  • // ------------------------------------------------------ //
  • // ------------------------------------------------------ //
  •  
  •  
  • // Now that we have our pre-signed URL, we can use it to upload a file to Amazon S3.
  • // Note that the pre-signed URL is specific to a given file and expiration date. As
  • // such, this doesn't grant free-range; but, rather very targeted access based on
  • // both the resource key and the file type.
  • uploadRequest = new Http(
  • method = "put",
  • url = preSignedUrl,
  • getAsBinary = "yes"
  • );
  •  
  • // This header is required if the contentType is non-empty in the signature.
  • uploadRequest.addParam(
  • type = "header",
  • name = "Content-Type",
  • value = "image/jpg"
  • );
  •  
  • uploadRequest.addParam(
  • type = "body",
  • value = fileReadBinary( expandPath( "./monkey.jpg" ) )
  • );
  •  
  • result = uploadRequest.send();
  •  
  • // Output the results of the pre-signed URL upload.
  • writeOutput( result.getPrefix().statusCode );
  • writeOutput( "<br />" );
  • writeOutput( charsetEncode( result.getPrefix().fileContent, "utf-8" ) );
  •  
  • </cfscript>

As you can see, the URL of the HTTP PUT is the pre-signed URL. And, when we run the above code, we get the following CFDump output:

200 OK

The image binary was successfully upload to Amazon S3 and resides at the resource defined in the pre-signed URL. By default, the uploaded object is "private"; if you want to make it public, you have to add additional Amazon Headers to the request (though, I have not tried this personally).

While the Amazon S3 upload policy allows for a lot more flexibility, uploading files using a pre-signed URL is certainly much more straightforward. Now that I see this, I'll have to go back and try to refactor some of my "upload policy" experiments.




Reader Comments

@All,

So, last night, I tried to play around with HTTP Form posts to Amazon S3 using pre-signed URLs; but, it looks like that isn't supported. In some older documentation for POST, I found this:

> Query string authentication is not supported for POST.

And, it seems rather complicated to get PUT to work from the browser.

That said, it seems like a good use-case for the PUT-based upload with pre-signed URLs is if you need to pass an upload URL to some other server-side process that needs to upload a file but doesn't necessarily have access to your credentials (such as a Message Queue job).

Reply to this Comment

Very helpful example!
I was relying on <cffile> operations and was experiencing an issue where .jpg files were uploaded with a metadata content type of application/octet.
This example contained the content type definition I was looking for.

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.