Zipping Image Archives With DEFLATE And STORE Compression Methods In Lucee CFML 5.2.9.31
For the last few years, one of my teammates - David Bainbridge - has been suggesting that we switch our zipping / archiving algorithms over to use STORE instead of DEFLATE when creating an archive of images. The idea being that most images file-formats are already compressed; which means that attempting to compress the images further during the zip operation does nothing but use unnecessary CPU. Now my team is planning to add some new zipping functionality, I thought it would be a good time to start looking at the difference between DEFLATE and STORE in Lucee CFML 5.2.9.31.
Traditionally, ColdFusion has not offered the ability to set any type of compression method in its various zipping features - it just uses DEFLATE, and that's the end of it. That said, it looks like Zac Spitzer recently added compressionMethod to the CFZip tag in Lucee CFML 5.3.3.x. So that's very exciting!
Of course, as luck would have it, I'm still on Lucee CFML 5.2.x at work; so, in order to experiment with different storage compression methods, I'm going to use the zip package that you can install via apt-get.
To get started, I created a simple Dockerfile that uses the Ortus Solutions CommandBox Docker image and installs the zip binary using apt-get:
FROM ortussolutions/commandbox:lucee5
RUN apt-get update && \
apt-get install -y \
zip && \
apt-get clean
Then, I created a docker-compose.yml file to bring this experimental Lucee CFML container online:
version: "2.4"
services:
cfml:
build: "." # Build our Dockerfile.
ports:
- "8080:8080"
volumes:
- "./:/app"
environment:
cfconfigfile: "/app/.cfconfig.json"
APP_DIR: "/app/wwwroot"
healthcheck:
test: "echo hello"
Once I had this Lucee CFML container running with the zip binary installed, I went about testing the different compression methods. The test is simple: I have a directory of images (about 17Mb worth); and, I create an archive of the images, once using the default compression method, DEFALTE, and once using the compression method, STORE.
ASIDE: I'm relatively new to the
zipbinary. As such, getting this to work properly with theCFExecutetag required a good deal of trial-and-error. I used themanpage extensively, supplemented by various StackOverflow threads. For reasons unclear to me, executing thezipbinary from the server terminal resulted in a different behavior when compared to theCFExecutetag, even when all the paths in the command were absolute.
<cfscript>
// Reset demo on subsequent executions.
cleanupFile( "./images-zip.zip" );
cleanupFile( "./images-zip-0.zip" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// First, let's try archiving the images using the DEFAULT COMPRESSION settings.
timer
type = "outline"
label = "Default ZIP Settings"
{
execZip([
"-r", // Recurse the input directory.
"-j", // Junk file paths (only store filenames, resulting in flat directory).
expandPath( "./images-zip.zip" ), // Output file.
expandPath( "./images" ), // Input directory.
"-x *.DS_Store" // Don't include files in zip.
]);
}
echo( "<br />" );
echo( "Zip file size: " );
echo( numberFormat( getFileInfo( "./images-zip.zip" ).size ) & " bytes" );
echo( "<br /><br />" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// Second, let's try archiving the images using NO COMPRESSION (ie, just storing the
// files as an archive, but not attempting to reduce the file size at all). Since we
// are working with images, the contents are already compressed; in most cases, the
// zip algorithm won't be able to remove nay size.
timer
type = "outline"
label = "Zero-Compression ZIP Settings"
{
execZip([
// Regulate the speed of compression: 0 means NO compression. This is setting
// the compression method to STORE, as opposed to DEFLATE, which is the
// default method. This will apply to all files within the zip - if we wanted
// to target only a subset of file-types, we could have used "-n" to white-
// list a subset of the input files (ex, "-n .gif:.jpg:.jpeg:.png").
"-0",
"-r", // Recurse the input directory.
"-j", // Junk file paths (only store filenames, resulting in flat directory).
expandPath( "./images-zip-0.zip" ), // Output file.
expandPath( "./images" ), // Input directory.
"-x *.DS_Store" // Don't include files in zip.
]);
}
echo( "<br />" );
echo( "Zip file size: " );
echo( numberFormat( getFileInfo( "./images-zip-0.zip" ).size ) & " bytes" );
echo( "<br /><br />" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
/**
* I execute the zip command-line utility using the given arguments. Standard output
* is printed to page. If an error is returned, the page request is aborted.
*
* @zipArguments I am the command-line arguments for zip.
*/
public void function execZip( required array zipArguments ) {
execute
name = "zip"
arguments = zipArguments.toList( " " )
variable = "successOutput"
errorVariable = "errorOutput"
timeout = 10
terminateOnTimeout = true
;
if ( len( errorOutput ?: "" ) ) {
dump( errorOutput );
abort;
}
echo( "<pre>" & ( successOutput ?: "" ) & "</pre>" );
}
/**
* I delete the given file if it exists.
*
* @filename I am the file being deleted.
*/
public void function cleanupFile( required string filename ) {
if ( fileExists( filename ) ) {
fileDelete( filename );
}
}
</cfscript>
In the second call to the CFExecute tag, I am passing in -0 as the first command-line argument. This moderates the compression speed, where -0 tells zip to use the compression method, STORE. This will apply to all files in the zip - if we wanted to be more targeted, we could have used the -n command-line argument to apply STORE to a set of white-listed file-extensions.
That said, if we run the above Lucee CFML code, we get the following browser output:
As you can see, when using the STORE algorithm (via the -0 compression speed setting), the archive of images is generated significantly faster with essentially the same file-size. Of course, the speed of execution is going to vary a bit on each page request; however, the STORE method was consistently about twice-as-fast as the DEFALTE method.
It looks like David Bainbridge was right - using the STORE method when creating image archives is going to be the smart choice. I'm excited to see that compressionMethod was added to the recent releases of Lucee CFML. However, since I'm still on an older version of Lucee, at least I can fallback to the zip binary; or, who knows, maybe dive down into the Java layer.
Want to use code from this post? Check out the license.
Reader Comments
You can also create tar archives using Lucee's compress() function, but the bundled version of apache commons compress, 1.9 from 2016 is a bit old and misses out on some of the improvements in the last 4 years.
https://docs.lucee.org/reference/functions/compress.html
https://github.com/lucee/Lucee/blob/5.3/core/src/main/java/lucee/commons/io/compress/CompressUtil.java
@Zac,
To be honest, I have no idea what a tar file is :D This may be a silly question, but can any [common] computer open up a tar file? For example, if I was generating a zip/tar file for the user to download it, would double-clicking on it work for them? Or do you have to be more tech-savvy to have tar capabilities?
@All,
When I was looking at using
zipwithCFExecute, one of the hurdles is that you cannot tellCFExecuteto use any particular working directory. And, unfortunately, the way to get relative folder paths in the resultant archive is to use a working directory in combination with a relative folder input. As such, I wanted to revisit this approach using Java'sProcessBuilder:www.bennadel.com/blog/3810-executing-command-line-processes-from-a-working-directory-using-processbuilder-in-lucee-cfml-5-2-9-31.htm
I had never heard of the
ProcessBuilderclass until Brad Wood mentioned it on Twitter a few weeks back (during my GraphicsMagick exploration, I think). It seems pretty cool; and affords us ways to set working directories, define environment variables, and manipulate inputs and outputs.@All,
I just realized that I wasn't properly scoping my local variables in the
exeZip()method. The success + error variable names should have been prefixed withlocal.. So, currently it is:... and it should be:
Without the
local., the variables get stored in thevariablesscope, not thelocalscope.As an aside, you can always set the
localModeof the function tomodernif you want to change the way unscoped variable assignments work:www.bennadel.com/blog/3678-using-function-localmode-modern-to-more-safely-render-coldfusion-templates-in-lucee-5-3-2-77.htm
@All,
As a small follow-up to this, I wanted to look at using both the
STOREandDEFLATEcompression methods in a singlezipcall:www.bennadel.com/blog/3888-using-both-store-and-deflate-compression-methods-with-the-zip-cli-in-lucee-cfml-5-3-6-61.htm
This allows a directory to be recursively compressed with selective-application of compression based on matching file-extensions.