Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
Ben Nadel at CFinNC 2009 (Raleigh, North Carolina) with: Dan Wilson
Ben Nadel at CFinNC 2009 (Raleigh, North Carolina) with: Dan Wilson@DanWilson )

Using The Amazon Web Services (AWS) SDK To Create Pre-Signed S3 URLs With Path-Style Access

By Ben Nadel on
Tags: ColdFusion

Amazon's Simple Storage Service (S3) provides two different ways to structure your S3 URLs. In the "virtual-hosted-style", your bucket name becomes part of the domain; and, in the "path-style", your bucket name becomes part of the resource (as a prefix to your object key). When you're making requests over HTTP, these two approaches are equivalent. However, when you start making requests over HTTPS (with SSL), things can get complicated. As such, it can be important to be able to tell your Java S3 Client to produce pre-signed URLs using path-style access.

Amazon S3 supports SSL; however, the SSL certificate only supports one-level of wildcard. So, for example, the SSL certificate for the US Standard / US-East-1 region supports:

*.s3.amazonaws.com

This means that if your S3 bucket name is going to be part of the domain name, it cannot contain any periods. My bucket - testing.bennadel.com - therefore cannot use the virtual-hosted-style access and support SSL at the same time. Doing so will result in a browser security exception:


 
 
 

 
 Amazon Web Services (AWS) SSL error for virtual-hosted-style S3 domains. 
 
 
 

As such, it's critical that I get my Java S3 Client to produce pre-signed URLs with path-style access. To do this, I have to use the .setS3ClientOptions() on the AmazonS3Client. It took me a while to figure this out, so I wanted to share the code that makes it happen.

First, I have to set up the context for the demo. In this case, I'm going to be using ColdFusion's (10+) ability to load custom JAR files ino the class-loader. I am not sure that I actually need all of the JAR files that come with the Amazon Web Service's (AWS) Software Developer's Kit (SDK) for Java; but, I find that I get object instantiation errors if I don't include at least some of them:

  • component
  • output = false
  • hint = "I define the application settings and event handlers."
  • {
  •  
  • // Setup the core application settings.
  • this.name = hash( getCurrentTemplatePath() );
  • this.applicationTimeout = createTimeSpan( 0, 0, 1, 0 );
  • this.sessionManagement = false;
  •  
  • // Here, we're adding the Amazon SDK for Java to the Java libraries that this
  • // application knows about. This will make the SDK classes available to the
  • // createObject() method.
  • // --
  • // NOTE: I am not sure that I actually need all of these classes; but, I get object
  • // instantiating errors if I don't include some of them. As such, I'm just including
  • // all of the JAR files that came with the AWS SDK download.
  • this.javaSettings = {
  • loadPaths = [
  • "../aws/lib/",
  • "../aws/third-party/aspectj-1.6/",
  • "../aws/third-party/commons-codec-1.6/",
  • "../aws/third-party/commons-logging-1.1.3/",
  • "../aws/third-party/freemarker-2.3.18/",
  • "../aws/third-party/httpcomponents-client-4.3/",
  • "../aws/third-party/jackson-annotations-2.3.0/",
  • "../aws/third-party/jackson-core-2.3.2/",
  • "../aws/third-party/jackson-databind-2.3.2/",
  • "../aws/third-party/javax-mail-1.4.6/",
  • "../aws/third-party/joda-time-2.2/",
  • "../aws/third-party/spring-3.0/"
  • ],
  •  
  • // Allows createObject() to access libraries in the main ColdFusion class path.
  • loadColdFusionClassPath = true
  • };
  •  
  •  
  • // ---
  • // PUBLIC METHODS.
  • // ---
  •  
  •  
  • /**
  • * I configure the application on startup.
  • *
  • * @output false
  • */
  • public boolean function onApplicationStart() {
  •  
  • var baseDirectory = getDirectoryFromPath( getCurrentTemplatePath() );
  •  
  • var config = deserializeJson( fileRead( "#baseDirectory#../config.json" ) );
  •  
  • // Make the AWS credentials globally available for the demo.
  • application.aws = {
  • region = config.aws.region,
  • bucket = config.aws.bucket,
  • accessID = config.aws.accessID,
  • secretKey = config.aws.secretKey
  • };
  •  
  • // Return true so the application can load.
  • return( true );
  •  
  • }
  •  
  • }

This configuration allows me to create the AWS Java classes using ColdFusion's createObject("java") function. Now, let's take a look at using the AmazonS3Client to generate pre-signed URLs. In the following code, I'm going to create two pre-signed URLs to the same S3 object. One will be generated before I call .setS3ClientOptions(); the other will be generated after I call .setS3ClientOptions(), specifically telling the S3 client to use path-style access.

  • <cfscript>
  •  
  • // Set up the S3 client with the explicit credentials.
  • s3Client = createObject( "java", "com.amazonaws.services.s3.AmazonS3Client" ).init(
  • createObject( "java", "com.amazonaws.auth.BasicAWSCredentials" ).init(
  • javaCast( "string", application.aws.accessID ),
  • javaCast( "string", application.aws.secretKey )
  • )
  • );
  •  
  • // Set the region (ex, "us-east-1") for the client. This will determine the endpoint
  • // that the client will use to make API calls and pre-signed URLs.
  • s3Client.setRegion(
  • createObject( "java", "com.amazonaws.regions.Regions" ).fromName(
  • javaCast( "string", application.aws.region )
  • )
  • );
  •  
  • // Create the local (server time) for Epoch. This will make it easier to calculate
  • // the expiration dates for the pre-signed URLs.
  • localEpoch = dateConvert( "utc2local", "1970/01/01" );
  •  
  • // Setup the expiration date for the pre-signed URLs.
  • expiresAt = dateAdd( "h", 1, now() );
  •  
  •  
  • // ------------------------------------------------------ //
  • // ------------------------------------------------------ //
  •  
  •  
  • // By default, the S3 client will use "virtual-hosted-style" URLs in which the bucket
  • // name is part of the domain name used to define the URLs. This works fine if you
  • // are generating HTTP URLs; but, if you intend to generate HTTPS (SSL) URLs (which
  • // the client does by default), you can run into issues if your bucket name contains
  • // periods (ex, "testing.bennadel.com). Amazon S3 only has a single wild-card SSL
  • // certificate; as such, it can't validate for multi-level sub-domains.
  • presignedUrl = s3Client.generatePresignedUrl(
  • javaCast( "string", application.aws.bucket ),
  • javaCast( "string", "sdk/pre-signed-urls/calm-face.jpg" ),
  • createObject( "java", "java.util.Date" ).init(
  • javaCast( "long", ( dateDiff( "s", localEpoch, expiresAt ) * 1000 ) )
  • )
  • );
  •  
  • // Will use "https://testing.bennadel.com.s3.amazonaws.com/" as domain.
  • writeOutput( presignedUrl & "<br />" );
  •  
  •  
  • // ------------------------------------------------------ //
  • // ------------------------------------------------------ //
  •  
  •  
  • // We can tell our S3 client to forgo to the "virtual-hosted-style" URLs and use
  • // the "path-style" access. With path-style access, the bucket name is part of the
  • // path used to define the resource (it is the key prefix). This will use your
  • // region-specific Amazon S3 endpoint (ex, s3.amazonaws.com) as the domain, which
  • // will allow the URLs to work with the Amazon S3 certificate.
  • s3Client.setS3ClientOptions(
  • createObject( "java", "com.amazonaws.services.s3.S3ClientOptions" )
  • .withPathStyleAccess( javaCast( "boolean", true ) )
  • );
  •  
  • // Generate the pre-signed URL (again).
  • presignedUrl = s3Client.generatePresignedUrl(
  • javaCast( "string", application.aws.bucket ),
  • javaCast( "string", "sdk/pre-signed-urls/calm-face.jpg" ),
  • createObject( "java", "java.util.Date" ).init(
  • javaCast( "long", ( dateDiff( "s", localEpoch, expiresAt ) * 1000 ) )
  • )
  • );
  •  
  • // Will use "https://s3.amazonaws.com/" as domain for us-east-1 region.
  • writeOutput( presignedUrl & "<br />" );
  •  
  • </cfscript>

When I run this code, I get the following output (truncated for readability):

https://testing.bennadel.com.s3.amazonaws.com/sdk/pre-signed-urls/....

https://s3.amazonaws.com/testing.bennadel.com/sdk/pre-signed-urls/....

As you can see, the first URL includes my bucket - testing.bennadel.com - as part of the domain name. This will result in an SSL error. The second URL, however, includes my bucket name as part of the resource path. This second URL can safely use SSL, regardless of bucket name.

Amazon S3 buckets make for a funky topic. I feel like they have a number of caveats that you don't truly consider until after you've created them and have started integrating S3 into your application. Then, at that point, it's often "too late" to do much about it. Thankfully, the Amazon S3 Client for Java allows us to use both types of object access for those of us that [unfortunately] have periods in our bucket names.




Reader Comments

I'm trying to implement the AWS SDK to utilize their Cognito service for one of our mobile application. However, when I was testing your code for initializing the AmazonS3 service I get the following error.

Error casting an object of type java.lang.NoClassDefFoundError cannot be cast to java.lang.Exception to an incompatible type. This usually indicates a programming error in Java, although it could also mean you have tried to use a foreign object in a different way than it was designed.

I'm current using Java jdk1.8.0_40 on CF10. Any help would be much appreciated.

Reply to this Comment

@Jae,

Very interesting problem. To be honest, the way that ColdFusion interacts with the Java classes is a bit of mystery to me. For example, when you load the JAR files using the per-application settings (as in my post), there is an option `loadColdFusionClassPath` which deals with how the classes can be accessed. I don't fully understand what that does or what the implications are for that. But, setting that to `false` might help with your issue.

I'd just caution about removing the jets3t library, since I believe that is what ColdFusion uses to do their S3 integration (such as with file IO operations, s3://). But, if you're using the SDK for all the AWS interactions, then I guess you'll be OK.

I wish I had a better handle on how Java really worked.

Reply to this Comment

@Ben,

I am so confused. I decided to put the jets3t-0.8.1.jar file back to /cfroot/lib/ directory and set the option 'loadColdFusionClassPath' to false. Afterwards, I restarted the CF service. I no longer receive an error. However, I decided to set the 'loadColdFusionClassPath' to true and still no error. Not sure what caused my original error. In conclusion, I have no idea what I did to make it work or how the error was being caused. I'm even more confused.

Reply to this Comment

@Jae,

Ha ha, sometimes, just restarting ColdFusion fixes things :D Also, in the ColdFusion admin, there are some settings about caching class files. I am not sure how or if those settings might affect the Java class paths. I will say that in my experience, Java paths do seem to get cached heavily some times.

When in doubt, restart ;)

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.