At InVision, we recently performed a large infrastructure upgrade. And, after we did this, we noticed that image thumbnailing times and CPU-load shot through the roof. Suddenly, we were seeing massive CPU usage (80%-100%) and thumbnail times that took up to 45 minutes on average! It didn't seem to affect all images; but, when it did, it slowed down everything on the machines.
When this was happening, all of the stack traces looked like this:
- locked <0x6fcba56b> (a sun.java2d.cmm.kcms.ICC_Transform)
After a good deal of Googling, I was able to find some blog posts and forum threads that seemed to be talking about the same problem (although, I am not entirely sure it's all about the same issue - my Java chops are noobish):
- JVM is stuck inside cmmColorConvert
- Resample image - drawImage exception - Java
- 20x slowdown in PNG processing when switching from JDK 1.6.0_17 to 1.6.0_18
- IV19562: CRASH IN JAVA_SUN_AWT_COLOR_CMM_CMMCOLORCONVERT() IN LIBCMM.SO
From what I can piece together here, it seems like certain image operations (such a resizing) on certain PNG images require color profiles to be converted. And, there's a bug in the JDK that doesn't handle this operation well when done under high load. One of the links above talks about it being a concurrency problem and a synchronization bug in the JDK.
In one of the other threads, one of the participants talks about why color profiles might need to be changed on the fly:
In your case, the encoded color model seems to differ from the original one which is why the ImageEncodingHelper (from XML Graphics Commons) falls back to encoding sRGB data which has to go through color management. And that's what's making the process very slow. Every pixel is converted to sRGB by the color profile associated with the BufferedImage that has been built from the PNG file.
So, it seems like there might be an already-slow process that is being further aggravated by a concurrency problem in the color management module in the JDK that falls-over under load.
We're currently running on ColdFusion 10; but, it sounds like its a problem with the underlying Java libraries, not with ColdFusion itself. And, I wish I could tell you that I found a way to fix this in ColdFusion; but, I didn't. We ended up converting [almost] all of our image thumbnail processing into ImageMagick operations on the command-line. Which, by the way, works beautifully!
Anyway, no solution here; but, if others run into the same problem, hopefully this data can help point them into the right debugging direction.