In my earlier post on working around breaking changes in RegEx matching in Adobe ColdFusion (ACF) 2025 update 14, I mentioned that normalizing line-endings is a best practice. When you normalize line-endings to be the single character newline (\n), it unlocks several advantages:
-
It makes your regular expression patterns easier to read and write since you don't have to account for
\r\n?variations. -
It prevents
\rfrom ever being consumed by a regular expression when the multiline flag ((?m)) is enabled (since there is no\rto consume). -
It allows you to treat multiline content as a newline-delimited list since you know that
\nis the only delimiter. This allows you to bypass regular expressions altogether in some scenarios.
I've talked about normalizing special character variations in ColdFusion before; but in the years since then, I've also added ACF-specific polyfills for bracket-based field grouping (ex, ids[]) that work with form scope data, url scope data, and their multipart/form-data counterparts. Which means that request data can now arrive "at the server" in complex objects, not just simple strings.
In my earlier forms of text normalization, I looped over the form scope and updated the values in-place:
for ( key in form ) {
form[ key ] = normalizeText( form[ key ] );
}
But now that I have to contend with arrays, it gets a bit more complicated. And, because I'm a programmer doing this stuff for funzies, I made it even more complicated than it had to be by going full recursion!
Before we look at the code, though, let's consider the division of responsibility. I've broken it up into three separate components because they each focus on different things:
-
Field-Grouping Polyfill - this is a platform level polyfill. Which means that I shouldn't even need it; and my application only carries it until Adobe finally implements it natively. As such, this polyfill is applied to every request via the
onRequestStart()life-cycle method of theApplication.cfcColdFusion framework component. Meaning, it's not a decision to be made by my application — it's a fact-of-life that I'm asserting. -
Scope Normalizer - this is an opinionated approach for a specific area of my application: the blog and comment authoring. I'm performing a lossy mutation of the request data. As such, I don't apply scope normalization to every request by default (like I did with the polyfill). Instead, I apply scope normalization at the top of my router ingress.
This allows "scribble files" and other transient non-routed files to bypass the scope normalization. That said, I don't feel super strongly about this; and could be persuaded to apply scope normalization to every request.
-
Text Normalizer - this is an opinionated approach to special character normalization. It's consumed by the scope normalize; and is therefore only applied to the router-based requests.
Since the field-grouping polyfill is a "fact of the platform" and the scope/text normalization is an "opinion of the application", the scope normalizer has to assume that the request scopes (form and url) might show up as a mixture of complex and simple data. Hence the aforementioned recursion.
With that said, let's look at how my ColdFusion application processes requests. All routed requests flow through a root ingress, which is just a series of nested switch statements and cfmodule tags. This choke point creates a natural place to apply "application-wide" semantics.
This is a heavily truncated look at my router ingress. What I'm showing you is that I call the ScopeNormalizer.cfc once for each of the url and form scopes:
<cfscript>
// Define properties for dependency-injection.
scopeNormalizer = request.ioc.get( "core.lib.web.ScopeNormalizer" );
// ColdFusion language extensions (global functions).
include "/core/cfmlx.cfm";
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
try {
// Trim values and standardize ASCII variations. This is an opinionated choice and
// should be scoped (no pun intended) to the routing, not to the entire ColdFusion
// application context.
scopeNormalizer.normalizeScope( url );
scopeNormalizer.normalizeScope( form );
// .... rest of routing logic .... //
// .... rest of routing logic .... //
// .... rest of routing logic .... //
// NOTE: Since this try/catch is happening in the onRequest() event handler, we know
// that the application has, at the very least, successfully bootstrapped and that we
// have access to all the application-scoped services.
} catch ( any error ) {
// .... rest error handling logic .... //
}
</cfscript>
The ScopeNormalizer.cfc then recursively iterates over the given CFML scope and applies the text normalization to every value that's a strict string (ie, instance of java.lang.String).
Generally speaking, every simple value that's posted to the server in either the url scope or the form scope comes through as a string. The only reason that I'm using a strict comparison is because I might want to apply this same workflow to JSON-based payloads (which can contain non-string simple values like Booleans and numbers); and I don't want to coerce those values to strings accidentally. This is also the reason the my recursion handles both arrays and structs despite the fact that my field-grouping polyfill only generates arrays.
In the following code, note that I'm performing the text value normalization in-place since I have to work within the reference to the existing CFML scope:
component hint = "I provide a way to recursively normalize request scopes." {
// Define properties for dependency-injection.
property name="textNormalizer" ioc:type="core.lib.util.TextNormalizer";
// ColdFusion language extensions (global functions).
include "/core/cfmlx.cfm";
// ---
// PUBLIC METHODS.
// ---
/**
* I recursively walk the given scope, normalizing strings along the way. Arrays (and
* structs for completeness) are left intact, with values being normalized in-place.
*
* Note: this is assumed to run AFTER the field-group polyfill has been applied, which
* is the primary way by which a complex object would be made available in the scope.
*
* Todo: in the long term, I can come up with a way to opt-out of this, probably with
* some sort of sibling "noNormalize" field; but for now, I'm assuming that all string
* values should be normalized.
*/
public void function normalizeScope( required struct cfmlScope ) {
// Since the normalizeValue() method normalizes complex objects in-place, we can
// just pass the scope directly to it for recursive normalization.
normalizeValue( cfmlScope );
}
/**
* I recursively walk the given value, normalizing strings along the way. Arrays (and
* structs for completeness) are left intact, with values being normalized in-place.
*
* Note: since this is expecting to work with HTTP request data, it has no safe-guards
* for null values. Empty form and url values are submitted as strings, even when using
* the `name[]` notation polyfill.
*/
public any function normalizeValue( required any value ) {
// Caution: this is doing a STRICT INSTANCE CHECK of the value against the Java
// String class. This way, we don't try to normalize castable values (like dates
// and numbers). That said, in a HTTP-delivered scope (url and form), it's
// essentially true that all simple values are strict strings until you get into
// other types of request payloads, like JSON.
if ( isString( value ) ) {
value = textNormalizer.normalizeAll( value );
} else if ( isArray( value ) ) {
value.each(( subvalue, i ) => {
value[ i ] = normalizeValue( subvalue );
});
} else if ( isStruct( value ) ) {
value.each(( key, subvalue ) => {
value[ key ] = normalizeValue( subvalue );
});
}
return value;
}
}
For the scope normalization, I'm calling .normalizeAll() because I want to apply "all the things"; but internally to the TextNormalizer.cfc, the normalization algorithm is broken up into two semantic buckets:
-
normalizeStructure()- this normalizes the representation of the text without changing its meaning. This does things like stripping-out zero-width characters, decomposing sequences into NFC (Normalization Form C) composition, and standardizing whitespace. Essentially, running this method on the given value shouldn't result in any visible changes to the rendered text. -
normalizeTypography()- this is the opinionated normalization of text content. It performs a lossy mutation, standardizing lots of things that "look the same(ish)" into a single representation. For example, it turns "smart quotes" in normal quotes; and it turns various "markers" into a simple bullet point (*).
The normalizeTypography() method almost certainly does "the wrong thing" in some edge-cases. For example, if a reader on this blog were to attempt to post a comment explicitly talking about the Black Leftwards Bullet, any attempt to post character \u204c in their comment text would be thwarted; and their bullet would land in the database as *.
Like I said, this is an opinionated approach to normalization. And I'm erring on the side of simplifying content and, especially, fixing copy-paste wonkiness rather than worry about edge-cases.
Here's my TextNormalizer.cfc. Note that it makes heavy use of regular expressions which, as always, are the freakin' bee's knees!
component hint = "I provide methods for normalizing special characters within text values." {
// ColdFusion language extensions (global functions).
include "/core/cfmlx.cfm";
// --
// PUBLIC METHODS.
// --
/**
* I apply all the normalization methods to the given value.
*/
public string function normalizeAll( required string value ) {
value = normalizeStructure( value );
value = normalizeTypography( value );
return value;
}
/**
* I apply representation-level normalization methods to the given value: changes the
* encoding of the text, not its content or intent.
*/
public string function normalizeStructure( required string value ) {
// Caution: these two methods must be run first and in this order such that they
// set up the rest of the normalization methods for success (re: pattern matching
// and text structure).
value = normalizeZeroWidth( value );
value = normalizeComposition( value );
// --
value = normalizeLineEndings( value );
value = normalizeSpaces( value );
value = normalizeEmptyLines( value );
return value.trim();
}
/**
* I apply content-level normalization methods to the given value: rewrites characters
* the author chose, which I am now un-choosing for them.
*/
public string function normalizeTypography( required string value ) {
// Note: none of these methods depend on normalizeStructure() - specifically the
// cautionary note about zero-width and composition ordering. As such, it's safe
// to normalize typography with-or-without the structural changes.
value = normalizeDoubleQuotes( value );
value = normalizeSingleQuotes( value );
value = normalizeDashes( value );
value = normalizeBullets( value );
value = normalizeEllipsis( value );
return value;
}
// --
// PRIVATE METHODS.
// --
/**
* I use Java's Pattern engine to perform a RegEx replace on the given input.
*/
private string function jreReplace(
required string input,
required string pattern,
string replacement = ""
) {
return javaCast( "string", input )
.replaceAll( pattern, replacement )
;
}
/**
* I replace special bullets with the standard asterisk.
*/
private string function normalizeBullets( required string value ) {
// Not using because it's often used to represent syllable breaks mid word.
// - \u2027 ## Hyphenation Point.
return jreReplace(
value,
"(?x)[
\u2022 ## Bullet.
\u2023 ## Triangular Bullet.
\u2043 ## Hyphen Bullet.
\u204c ## Black Leftwards Bullet.
\u204d ## Black Rightwards Bullet.
\u2219 ## Bullet Operator.
\u25a0 ## Black Square.
\u25a1 ## White Square.
\u25aa ## Black Small Square Emoji.
\u25ab ## White Small Square.
\u25cb ## White Circle.
\u25cf ## Black Circle.
\u25e6 ## White Bullet.
\u29be ## Circled White Bullet.
\u29bf ## Circled Bullet.
]",
"*"
);
}
/**
* I normalize the text to Unicode Normalization Form C (NFC). This composes
* canonically equivalent sequences into their single code-point form so that visually
* identical strings are byte-identical.
*/
private string function normalizeComposition( required string value ) {
var normalizer = new java( "java.text.Normalizer" );
var form = new java( "java.text.Normalizer$Form" );
return normalizer.normalize( javaCast( "string", value ), form.NFC );
}
/**
* I replace like-sized dashes with standard dashes.
*/
private string function normalizeDashes( required string value ) {
return jreReplace(
value,
"(?x)[
\u2010 ## Hyphen.
\u2011 ## Non-Breaking Hyphen.
\u2012 ## Figure Dash.
\u2013 ## En Dash.
\u2212 ## Minus Sign.
\ufe63 ## Small Hyphen-Minus.
\uff0d ## Fullwidth Hyphen-Minus.
]",
"-"
);
}
/**
* I replace "smart double quotes" with standard double quotes.
*/
private string function normalizeDoubleQuotes( required string value ) {
return jreReplace(
value,
"(?x)[
\u201c ## Left Double Quotation Mark.
\u201d ## Right Double Quotation Mark.
\u201e ## Double Low-9 Quotation Mark.
\u201f ## Double High-Reversed-9 Quotation Mark.
\u2033 ## Double Prime.
\u2036 ## Reversed Double Prime.
\u275d ## Heavy Double Turned Comma Quotation Mark Ornament.
\u275e ## Heavy Double Comma Quotation Mark Ornament.
\u2e42 ## Double Low-Reversed-9 Quotation Mark.
\u301d ## Reversed Double Prime Quotation Mark.
\u301e ## Double Prime Quotation Mark.
\u301f ## Low Double Prime Quotation Mark.
\uff02 ## Fullwidth Quotation Mark.
]",
""""
);
}
/**
* I replace "smart ellipsis" with plain-text ellipsis.
*/
private string function normalizeEllipsis( required string value ) {
value = jreReplace(
value,
"(?x)[
\u2026 ## Horizontal Ellipsis.
]",
"..."
);
value = jreReplace(
value,
"(?x)[
\u2025 ## Two Dot Leader.
]",
".."
);
value = jreReplace(
value,
"(?x)[
\u2024 ## One Dot Leader.
]",
"."
);
return value;
}
/**
* I trim lines that contain nothing but spaces.
*/
private string function normalizeEmptyLines( required string value ) {
return jreReplace( value, "(?m)^[\s&&[^\r\n]]+$", "" );
}
/**
* I convert all the line-breaks to NewLine characters.
*/
private string function normalizeLineEndings( required string value ) {
var newline = chr( 10 );
return jreReplace(
jreReplace( value, "\r\n?", newline ),
"(?x)[
\u2028 ## Line Separator.
\u2029 ## Paragraph Separator.
\u0085 ## Next Line.
]",
newline
);
}
/**
* I replace "smart single quotes" with standard single quotes.
*/
private string function normalizeSingleQuotes( required string value ) {
return jreReplace(
value,
"(?x)[
\u00b4 ## Acute Accent.
\u02bc ## Modifier Letter Apostrophe.
\u2018 ## Left Single Quotation Mark.
\u2019 ## Right Single Quotation Mark.
\u201a ## Single Low-9 Quotation Mark.
\u201b ## Single High-Reversed-9 Quotation Mark.
\u2032 ## Prime.
\u2035 ## Reversed Prime.
\u275b ## Heavy Single Turned Comma Quotation Mark Ornament.
\u275c ## Heavy Single Comma Quotation Mark Ornament.
\u275f ## Heavy Low Single Comma Quotation Mark Ornament.
\uff07 ## Fullwidth Apostrophe.
]",
"'"
);
}
/**
* I convert any special spaces to regular spaces.
*/
private string function normalizeSpaces( required string value ) {
return jreReplace(
value,
"(?x)[
\u00a0 ## No-Break Space.
\u1680 ## Ogham Space Mark.
\u2000 ## En Quad (space that is one en wide).
\u2001 ## Em Quad (space that is one em wide).
\u2002 ## En Space.
\u2003 ## Em Space.
\u2004 ## Thick Space.
\u2005 ## Mid Space.
\u2006 ## Six-Per-Em Space.
\u2007 ## Figure Space.
\u2008 ## Punctuation Space.
\u2009 ## Thin Space.
\u200a ## Hair Space.
\u202f ## Narrow No-Break Space.
\u205f ## Medium Mathematical Space.
\u3000 ## Ideographic Space.
]",
" "
);
}
/**
* I strip out zero-width characters so they don't cause surprises.
*/
private string function normalizeZeroWidth( required string value ) {
return jreReplace(
value,
"(?x)[
\u00ad ## Soft Hyphen.
\u200b ## Zero Width Space.
\u2060 ## Word Joiner.
\ufeff ## Zero Width No-Break Space.
]",
""
);
}
}
The category of changes and the list of characters therein is likely incomplete. But so far, this is what's been working well for me and this site. Keep in mind that this site deals primarily with "human thoughts"; which are highly squishy and resilient to some lossy mutations.
Reader Comments
Post A Comment — ❤️ I'd Love To Hear From You! ❤️
Post a Comment →