Intermittent Bug In serializeJSON() In Adobe ColdFusion 2025
In one of the recent Adobe ColdFusion 2025 updates (maybe 7, maybe 8), I seem to be hitting a strange intermittent bug in the serializeJson() function. It only happens a handful of times a day; and in my recent debugging efforts, I've found that running a small sleep() and then re-trying the call seems to work. This is why I think it's a bug in ColdFusion itself and not in my code.
At the top of every request to my site, I generate a Content Security Policy (CSP) payload. Part of this payload includes a JSON-stringification call:
<cfscript>
var reportPayload = serializeJson({
group: "csp-endpoint",
max_age: 10886400,
endpoints: [
{
"url": reportToUrl
}
]
});
</cfscript>
There's nothing request-specific in this payload. The reportToUrl is a configuration value that never changes. And my site gets hits thousands of times a day with no problem. Except, on 5-6 requests, this serializeJson() call throws this nonsensical error:
Invalid argument value for
serializeJSON.The
SerializeQueryargument can be a boolean or string type only.
After a bunch of failed debugging steps — assuming it was my fault — I finally tried adding a sleep(100) and a retry. In the following code, notice that the serializeJson() call in each try block is the identical:
<cfscript>
var reportPayload = {
group: "csp-endpoint",
max_age: 10886400,
endpoints: [
{
"url": reportToUrl
}
]
};
try {
// ..... THIS CALL IS IDENTICAL TO NEXT ONE .....
var reportValue = serializeJson( reportPayload );
} catch ( any error ) {
logger.error( "Couldn't JSON CSP data (A).", { reportPayload } );
sleep( 100 );
try {
// ..... THIS CALL IS IDENTICAL TO PREV ONE .....
var reportValue = serializeJson( reportPayload );
} catch ( any error2 ) {
logger.error( "Couldn't JSON CSP data (B).", { reportPayload } );
rethrow;
}
}
</cfscript>
If the error were in my code, I would expect both the logger.error() calls to show up in Bugsnag. However, when I look at my logging after running this all day, here's what I get:
As you can see, only the (A) version of the logging is recorded. After the sleep(100), the repeated call to serializeJson() works without error; and the (B) version never shows up.
What I assume happened is that there must have been some sort of "security fix" introduced to the serializeJson() function which has inadvertently introduced a transient bug of its own. I will open a ticket and link it in the comments.
UPDATE: 2025-05-30
After publishing this yesterday, I went to create a "hotfix" method for serializeJson(). And, while writing that method, I wanted to see if maybe the sleep(100) wasn't actually necessary. So my hotfix method includes two fallback retries, one with a sleep, one without:
component {
/**
* HOTFIX: I provide a version of the serializeJson() method that runs mulitple
* attempts on failure. This is to patch an emergent bug in one of the latest ACF
* updates that seems to have introduced some timing wonkiness.
*/
public string function serializeJsonHotfix( required any input ) {
try {
return serializeJson( input );
} catch ( any error ) {
logger.info( "Serialize JSON hotfix (A)" );
}
try {
return serializeJson( input );
} catch ( any error2 ) {
logger.info( "Serialize JSON hotfix (B)" );
}
sleep( 100 );
try {
return serializeJson( input );
} catch ( any error3 ) {
logger.info( "Serialize JSON hotfix (C)" );
rethrow;
}
}
}
As you can see, (B) is just an immediate retry of the same serializeJson() call - no sleep at all. Then (C) is a second retry with a sleep(100). I deployed this version this morning and so far, all I've seen in the error logs are:
Serialize JSON hotfix (A)
Turns out that the immediate retry is sufficient to work around the bug. At least in my particular case.
Want to use code from this post? Check out the license.
Reader Comments
I've filed issue CF-4232120 in the Adobe Tracker.
I just posted an update that an immediate retry seems to be sufficient. But, the truth is, my log-line (posted above in the "Update") does its own
serializeJson()call internally to prepare the payload for Bugsnag. So it's possible that this internally call either:Provided enough delay.
Rejiggered the internal Java state enough.
Not sure. Either way, I'm now 1000% convinced this is a true ACF bug.
Interesting, as always, Ben. But as you've redacted what's in the logged output of A (in the catch), we're left to assume you've confirmed the value shown IS OK? If you tried to deserialize it on its own, it would work?
If not, do log B inside the nested try (BEFORE) it's nested deserializejson, just to ensure the results are identical.
If doing a manual deserialize of log A's result does work, that would indeed be odd, of course. But at that point I'd think you should consider just going ahead and doing the logging BEFORE the first deserialize (inside that first try).
Sure, that will be a lot of log lines, but then when you get the LOG A standing out, you can see how the log in that catch compares to the one that led to failure. They'd seemingly HAVE to be different. Or yes it's a very odd bug. :-)
I realize what I've written above may seem confusing to anyone just glancing at it quickly. If you have any doubts about what I'm proposing, Ben, I'd be happy to clarify. It seems worth doing.
Indeed, we can't try it for you ourselves since your code above (and in the bug report) doesn't show what you're passing in. And there may well be something to that (since again we can't see what it is), but doing this logging should confirm things either way for you.
Hope you'll consider it, even just to rule it out.
Oh, and if you're on Commandbox you could of course easily run different cf updates to confirm when things started failing. I get that it may fail for you only in prod, and you may prefer not to run older updates there. But perhaps some standalone test caller could replicate the issue. (If so, that would certainly help Adobe if it was not just about different inputs to the deserializatiin).
You may even find there's more to all this than meets the eye: if the logged results DO differ (when working vs when next failing), maybe some other change is causing THAT difference. And testing might even show that difference could have started due to some prior change, but is only exacerbated for you by a more recent one.
Sorry for the wall of text. Just thinking things out with you. As you know, I do this sort of thinking things through with folks daily in my troubleshooting consulting. And as I've offered before, if it would help you I'd happily work with you for free in a shared desktop session, in thanks for all you've done and do for the community.
Otherwise hope the above might help.
@Charlie,
All good thinking. And in fact, my thinking on this has been peeling back like an onion. At first, I thought it was the
sleep()that was making the difference. But then I realized that mylogger.()call would also be serializing the value that just failed to serialize. And that payload was showing up in Bugsnag.So the sleep wasn't the key. But then I thought maybe it was just the time-delay caused by the logging itself. So what I actually have running in production right now is more like:
... in this version (live right now), I now have back-to-back calls to the
serializeJson()with nothing in between - no sleep, no intermediary logging. And so far, the logging in the secondcatchblock hasn't shown up yet. And it's been live for several hours.I feel like this conclusively leads me to believe that there's literally some random error showing up in the
serializeJson()call; and that doing the call again works for unknown reasons.re: CommandBox - unfortunately, the error only showed up a handful of times a day, so it's hard to test. I have to try something, deploy, and wait. I could try to load-test it locally; but not sure it would surface anything.
@Ben Nadel, I understand. But you say, "But then I realized that my logger.() call would also be serializing the value that just failed to serialize."
In the original code, you don't show the logger call doing anything but logging the payload...and I assumed that would just log the json text.
And that's why I wanted to see if you'd compared what it was BEFORE the attempt to deserialize (when it would fail) to what it was when it did not. It COULD differ somehow.
But if you feel I'm somehow still barking up the wrong tree, we can let it go and see what Adobe or others may have to say, or what else you may find in time.
@Charlie,
I would say "wrong tree" only because I'm not actually mutating the payload in between JSON attempts. I'm just taking the same exact value that just failed and trying it again.
In the latter example, you could argue that maybe the value - which is being passed by reference - is mutated by the calling context in between the JSON attempts. It's not though. And, in my first incarnation, the
reportPayloadwhich is being logged is being generatedlocalto the Function call, so there was no calling context that was out of my control.You could also theorize that the
logger.()call was mutating the value. But then, in my final update, I have two back-to-backserializeJson()calls with no intermediary logging. And the second one works..... at least seems to - again, it only rarely errorred before (a few times a day).THAT SAID, the fact that I'm the only one who's run into this is definitely suspicious 🤪 Every ColdFusion app almost certainly uses JSON. But maybe it's rare for a JSON payload to be generated on every request (like I'm doing for my CSP policy).
All fair enough. I simply was asserting that you could/should prove via logging that the value was the same (within the catch, and before doing the deserialize there). Note that you were only doing it if that second one failed. But if you're still not persuaded, I'll let it go. :-)
@Charlie,
Sorry, I think maybe I just am not following what you're saying 🙃 I can at least tell you that payload that was showing up in Bugsnag is the payload that I was expecting - not values altered.
@Ben Nadel, OK. That was my first question. But in that first response I also offered something to try if that was indeed true. I still think it could help.
If you reread my first reply and now it makes sense, and you try my suggestion there, let us know what you find. If somehow it still does not (make sense to you), we can let it go.
PS Yes, I know you later said you are "not mutating" the value, but that doesn't mean it can't possibly be different before and after the two attempts to deserialize it. (Perhaps THIS is where the bug may present itself.) And I know you said you think it can't be mutated, but I've just been proposing you prove that rather than trust your instincts. :-)
This morning, I deployed this version - hopefully I won't see any more errors for the day:
This code is now doing nothing other than an immediate retry - no delay, no logging. If I don't see any errors, the I think it's further evidence that it's an Adobe bug itself.
If it does fail, there's still one other possibility beyond being a bug in serializejson--you could be experiencing just a classic race condition: it's POSSIBLE that the value of what you're passing to the serializejson COULD be different between the two calls. And it's even possible when "no time" has passed between them (as it's not really 0.000 ms).
And this is effectively what I was getting at from the start, in proposing that you could log that argument to the serializejson function both before each successful call AND before doing it within the catch, so that you could compare the two (voluminous though such logging could be).
And there are at least two apparent ways this could happen, looking at even that very simple code: first (less likely), note that you're not scoping that input argument. It seems clear you're expecting it to be found implicitly in the arguments scope--but the fact that you're not being explicit means that CF COULD technically find a variable named input in some other argument that earlier in the implicit scope search.
(Indeed, the fact that it's being found implicitly this way runs counter to the update from March 2024 which disabled that behavior by default. So either you are enabling it at either the jvm arg or app level, or perhaps you have simply not applied updates to this instance since then. I see it often, sometimes intentional, sometimes not.)
But a second (and perhaps more likely) issue allowing a potential race condition is that the "input" argument to the function is a struct. As you know, those are "passed by reference" in CF.
As such, if in the caller (to the method) the value passed in might itself come from some variable which could be somehow be changed in another request, then that change could affect the first call and yet be changed again by the time of the second call (even in just milliseconds).
And even though it's passed in as an argument to this method, that's still passed "by reference" in CF. It's not like it's somehow "frozen" once it's "in the arguments scope" of the method. (Lucee has added a new option in defining method args to use passBy="value", but CF has not. And while CF2021 added an app-setting to allow changing arrays to be pass-by-reference, it did not offer that for structs.)
The classic way to address this is of course to use the duplicate function, which makes a copy-by-value. I get some eschew it, but it doesn't HAVE to be expensive, if the complex var is rather small. And if you at least did that in your function, it would guarantee that the two serializejson calls would ALWAYS have the same values. That might be interesting (you shouldn't see it "fail then work" as you have above).
But that doesn't prevent a "bad" value making it into the method call/arguments scope, which would simply cause both calls to fail. Still, it would help confirm that there was no bug in serializejson but rather "just" a race condition in your app. It certainly can happen.
Sorry if it seems I'm beating a dead horse. I really am just helping think this through with you and other readers. I'll admit that when I hear "there's a bug", I am inclined to want think through whether there might be a different explanation. These seem reasonable ones, but even if somehow they prove NOT to be your issue, I hope this is perhaps helpful to someone.
Finally, I realized only now I kept referring to deserializejson when you were using serializejson. I had been responding over the weekend only on my phone. I'll blame the tiny screen. :-) And if I can edit the previous comments to correct that I will, so as not to confuse future readers. (Ah, no. I found out after posting this that the "edit this comment" is only offered for a comment one has just posted, not others.)
@Charlie,
Ok, I've added this to the top of every incoming request to the site. It's a static call to the
serializeJson()function. Nothing dynamic about it, and it's not using any references:Hopefully by the end of the day, I'll have at least one failure - and I think that will put the nail in the coffin.
@Ben Nadel, Good deal...as we were indeed running out of room for more nails on that coffin! :-) Let's see what happens.
Post A Comment — ❤️ I'd Love To Hear From You! ❤️
Post a Comment →