CFHTTPSession.cfc For Multi-CFHttp Requests With Maintained Session

Posted March 3, 2008 at 9:29 AM by Ben Nadel

Tags: ColdFusion

Reader Comments

dude, badass! I can see great potential for this kind of thing in programmatic, automated testing, too. Particularly "smoke test" kind of tests where you just want your tests to run through the site and make sure you don't get any 400/500 errors.

You should riaforge this!

Hey Ben

This is cool - is there a specific reason for using Firefox as the user agent?

Dom

Does this cfc handle cookies based on domain? I have a login that goes through several redirects between different servers, and through all the redirects, I only need to send the cookies based on the current domain. I have written something like this, but just a recursive function. It is not as clean as yours. Also does this handle the SSL certificates if I import them in the keystore?

Thanks, great post and source. This gives me a great example of coding, cause I´m still learning. Is there a special reason why to use Firefox as client?

@DrDom,

I use FireFox cause when I make a web call, I like to announce myself as the most awesome browser in town :)

@Matthew,

I never considered changing domains. I guess, this would just keep accumulating cookies and then send them across no matter what the next domain is. Since it manually follows the Location redirects sent back by CFLocation type tags, I think it will keep sending cookies.

Do you think I should make the cookies domain based? I could keep all the information keyed in a structure that is domain specific.

@Ray,

When I get some time, I will put it up.

No you dont have to worry about domain cookies. For my situation i just needed to hold onto the last set of cookies for that last redirect that was made.

I just dont want to send unecessary cookies. Once all the redirects happened, i just need the cookies from that final redirect.

Good job ben.

-Matthew

@Matthew,

Sounds good. If you see anywhere that this can be improved, let me know.

I was trying to write something like this the other day... couldn't figure it out and gave up. But THIS IS TOO COOL. Thanks for the lesson. Can't wait to try it out.

@Alan,

Glad you are excited about this. Let me know if you see any ways that it can be improved.

Coldfusion Server complained about the invalid token '{' in CFHTTPSession.cfc. Is it because we are running CFMX 6.1?

@Joshua,

Yeah, that's a version issue. This is ColdFusion 8 compatible code. The {} notation is an implicit struct. YOu can try to replace things like:

<cfset var LOCAL = {} />

... with:

<cfset var LOCAL = StructNew() />

There might be some other areas that are not compatible as well, but I believe that all of this should be able to be converted in ColdFusion MX 6 compatible.

Unfortunately, I don't have access to an MX6 machine and cannot test any of the code.

Okay. What about []? Put in ArrayNew(1)?

After I did just that, I am now looking at this piece of code where attribute, "array" is not supported.

<cfloop index="LOCAL.Param" array="#VARIABLES.Instance.RequestData.Params#">

@Joshua,

<cfloop index="LOCAL.Param" array="#VARIABLES.Instance.RequestData.Params#">

Becomes:

<cfloop
index="LOCAL.ParamIndex"
from="1"
to="#ArrayLen( VARIABLES.Instance.RequestData.Params )#"
step="1">

<!--- Get short hand to next param. --->
<cfset LOCAL.Param = VARIABLES.Instance.RequestData.Params[ LOCAL.ParamIndex] />

@Ben

I am now looking at the error where cfhttpparam's attribute, "attributecollection" is not supported.

Note: It has been few years since I last worked with ColdFusion. Much appreciated for your help!

@Joshua,

Oooh :( That's a bit of a tougher one! You have to take a little bit more code here to actually get that to work. Instead of just passing in the attribute collection, you are gonna have to define the individual CFHttpParam tags.

For example:

<cfcase value="Header">

<cfhttpparam
type="header"
name="#LOCAL.Param.Name#"
value="#LOCAL.Param.Value#"
/>

</cfcase>

<cfcase value="CGI">

<cfhttpparam
type="cgi"
name="#LOCAL.Param.Name#"
value="#LOCAL.Param.Value#"
encode="#LOCAL.Param.Encode#"
/>

</cfcase>

Basically, you have to enumerate each CFCase tag rather than just using the one tag.

@Ben

After putting in a bunch of code replacing a single line of attribute collection. I no longer get any compile error from CFHTTPSession.cfc.

I am now figuring out why I would get different JSESSIONID on every request against java/jsp off a tomcat. Am I supposed to see same JSESSION on every request?

Thanks.

@Joshua,

I don't know too much about jsessionID. I think you should keep it from page to page. Hmmm :( That's awesome that you got it to compile, though. Very well done.

@Joshua,

I have another person who is having trouble with maintaining session across pages. I am gonna try to debug that at lunch; I might find some good stuff. I will let you know.

Very good article.

When you call a cfhttp it will always open up a new request, unless you store the cookie info (jsession) in a session, or some kind of persistant state.

When i do these types of things with Cfhttp, i hold on to the object in a session so the cookies persist.

Hope that makes sense.

-Matthew

@Matthew,

The CFHTTPSession.cfc object should be passing back any cookies that it receives. It doesn't care what type of cookie they are - jsessionID, CFID, CFTOKEN, etc.

There must be something that is breaking.

I tested another site off different tomcat and noticed that it did return JSESSIONID cookie on the first request and did _not_ on the second request. Am assuming that it means it is carrying a session.

One thing I noticed is the difference in value for JSESSIONID between the first site (that fails) and the second site (that works). The first site returns something like "JSESSIONID=3609E7C2868A6A9D5136DBF8A61F592F.jvm1" while the second site would return about the same but without ".jvm1". I have not looked closely at the code or debugged yet but I wonder off head if the period (.) upsets CFHTTPSession?

@Joshua,

The (.) shouldn't do anything to upset the functionality, at least not that I can think of.

Just FYI, SalesForce.com has an extremely exhaustive API available, which doesn't require this sort of screen scraping. They offer it in various flavors, including WSDL, which should be extremely easy to consume from ColdFusion.

Screen scraping their site is against their terms of service, and if they found out it was happening, it could result in your client being charged back-fees as if they had purchased the API since the time you started screen scraping, or even worse losing their SalesForce account (which for most people I know who use SalesForce would essentially halt their business). Probably would depend on how much traffic you were running through there.

We had investigated doing this at my last job (which was a heavy SalesForce.com user), and decided the risk to the business was too much.

Not to belittle your component, it's still very cool, just warning you of the landscape; SalesForce are some pretty aggressive folks.

@Eric,

I appreciate the heads up. However, rest assured that I have talked to people from SalesForce.org and this is actually what they recommended that I do.

Actually, now that I look at what I said in the post, I misspoke. I was going to be doing a SalesForce.org integration, but the problem was not so much on their end as it was with another vendor. I need to login into this other vendor, run contact information and registration reports and then upload those to SalesForce.org using their API.

Sorry about the incorrect description of my problem :) I was more excited about the CFHttpSession.cfc and probably didn't proof read very well. I did run this by people as both SalesForce.org and with this third-party vendor and it was agreed that this screen scraping method was the best way to get the communication done.

Thanks for pointing out the error in my description.

BTW - I got the:

invalid token '{' in CFHTTPSession.cfc.

with CF 7.1 as well... Tried it on sever different servers. All the same. I will try the work-around discussed above to see if I can get it working tomorrow.

@Al,

The code is built for CF8. To see how to convert it into CF7 compatible code, check the comments above. I walked someone else through the process.

Thanks... I did spend about an hour with this thread last night trying to get my arms around it... I'm sure the problem is me... But I just couldn't get there... Too deep..

BUT - I do want to say that spending the time going through your code allowed me to break through the metal barrier I was having around how to maintain session state with CFHTTP. I did, in the end, solve my problem. Starting at your code helped me understand how to tease the cookie information out of the 'Set-Cookie' portion of the response-header. I had previously only ever used the FileContent portion, so this was very instructional. Once I saw how you were teasing the Set-Cookie info out in your CFC, I was able to reproduce that in my own code without the use of the CFC (which is really the understanding I lack)....

So, for anyone else who needs a solution to maintain session state between CFHTTP calls, but is not up to the task of using the CFC, there are a couple other ways around this... In the end, I am storing the cookie info in the database so I can pull it across different CF pages with different CFHTTP calls....

One other little trick I came across. I was getting "Cookies not enabled on your browser" for a while until I realized I needed to visit the site with a GET (rather than a POST) first, just for the purpose of seeing the cookie structure. Once you have the cookies, you can echo them back to the site on subsequent visits and all will be good.

Thanks again.. Very very helpful page here... There just isn't too much on this topic around the Internet that I could find...

Cheers....

@Al,

Glad that I could help out in some way, even if it was just for inspiration. Sounds like you have things moving along very nicely now.

I am trying to use your cfc to do a https login then switch to another page to do some scraping on the Pitney Bowes site in order to grab delivery information about our shipments. However it appears that the cfc is not functioning correctly for some reason.

I just get "Connection Failure" with nothing returned. I have debugging turned on in CF and see the page trying to call the cfc in its correct location, but the execution times are long, like 2500ms+

Does this cfc even support HTTPS ?

If so, any ideas on what the Connection Failure is being caused by?

I'm running CF8.

Oh, I have tried using the cfhttp tag but can't seem to get that to work either...

@Steve,

Hmm, I am not sure if I have tried HTTPS explicitly. It should work. I think it uses all the same cookies and stuff. If you cannot get straight up CFHTTP to work, then CFHttpSession definitely won't work.

Connection failure might have to do with the User Agent that is being used. The server might deny a given user agent... maybe? I am guessing. If you can get this connection failure on a public page, I can fool around with it.

I can log in with firefox 2.0.0.8 which is close to the 2.0.0.6 that you are using in the cfc. I have gotten your cfc to work on another site to log me in without https so I know I am using and coding it correctly.

I just can't get it to work on the pb.com site. Perhaps they are blocking login requests which do not originate from their website? If so, then I'm out of luck. I called their customer service yesterday and a guy said he would check and get back to me today, time will tell if they will give me any type of answer LOL

Is there any way to spoof a refering url via your cfc or cfhttp ?

@Steve,

I just looked at the code and I am a bit shocked that I dont' already have referer spoofing in there! Bananas! I will try to add that today.

Guys,

Ive had to add some additional headers to get connections to work over http(s) on certain domains.

I was receiving a connection failure as well.

Look at this and see if it helps.

http://www.talkingtree.com/blog/index.cfm/2004/7/28/20040729

I tried

<cfhttpparam type="Header" name="Accept-Encoding" value="deflate;q=0">
<cfhttpparam type="Header" name="TE" value="deflate;q=0">

and

<cfhttpparam type="Header" name="Accept-Encoding" value="*">
<cfhttpparam type="Header" name="TE" value="deflate;q=0">

in the cfhttp tag but they didn't make a difference. I also tried

.AddHeader( "Accept-Encoding", "*" )
.AddHeader( "TE", "deflate;q=0" )

in the cfc and no change. I'm really starting to wonder if they are denying this type of a login attempt as it is not coming from a page on their server.

@Steve,

I have update the CFHttpSession.cfc component to include automatic referral spoofing with possible manual overrides:

http://www.bennadel.com/index.cfm?dax=blog:1215.view

I have tested logging into PayPal.com and can properly log into HTTPS pages. Take a look and see if that helps.

I need to specify redirect="false" for the site I'm attempting to login to. I noticed that the default in the cfc is set to "no." Regardless, the header still returns a status-code of 200 rather than 302. Any clues for the utterly clueless? Thanks!

@Gernot,

The CFHttp grabs are set to ignore redirects... but that is because the redirect is then handled afterwards, programmatically. I am curious as to why you need to stop the redirect?

I'm submitting data collected from a pdf (livecycle) form to Coghead via their REST API . Because the login form redirects into the coghead UI, I need to stop the redirect to continue making requests via the api. I'm not sure why this is necessary but unless I specify redirect="false" as a cfhttp attribute, it appears that I lose my authentication token within the first call.

Since I'm only making a few requests to submit the data and I'm only receiving one cookie, I've found it to work to simply store the value in a variable then pass it back in each request via a named cfhttpparam tag. This is also interesting because I need to send it as a header rather than a cookie.

I'm thinking that I'm past most of these quirks and hope to be successfully creating records by the end of the afternoon (it's a massive form being submitted).

Thank you so much, Ben, for all of your examples esp. maintaining cfhttp sessions. It's a very timely resource that has taught me alot.

@Gernot,

Interesting problem. I am not sure how to advise on it. But, it looks like you are on the right track. Good luck.

If anyone's still wondering about the JSessionID with a period (.) in it --

I ran into a similar problem involving a hyphen (-) in a JSessionID. CF was urlencoding the hyphen to %2D, even if i specify encoded="yes". Same thing happens with the period going to %2E. I think the server I was trying to get to was expecting the hyphen to be unencoded. CF8 decodes the hyphen properly, though if I recall correctly, CF7 doesn't (unfortunately, that server recently died, so I can't easily check). Best solution I've found is to send the header manually, using Java's URLEncoder:

<cfset ue = CreateObject('java', 'java.net.URLEncoder')>
...
<cfhttpparam type="header" encoded="yes" name="Cookie"
value="#ue.encode(name)#=#ue.encode(value)#">

Not really sure why it seems CF's URLEncodedFormat doesn't use Java's URLEncoder, but oh well...

Excellent resource. This has probably saved me a few hours of searching. Thank you very much!

Is it possible to grab a third party's application session (non CF) and use that throughout the user's session?

For example, I managed to have it work with Moodle at initial login but after that Moodle knows that the sessions were not created yet so when a user clicks on say, their profile page, it forces them to login again.

Basically, we're building a SSO application.

How can I do this?

thanks

LN

Not that my comment will explain what causes the issue mentioned above. Just a comment that I am using CAS for SSO with Moodle and it works out nicely and all the more when you have multiple apps of different languages (php, dot net, coldfusion, etc) leveraging the same SSO.

Good Luck.

thanks Joshua.

We are a Novell shop and we are using iChain, a Novell product that has a SSO tool called Form Fill. We found out there is a but in iChain that chunks data during post/get of packets and so we get 504 timeout errors.

What is your CAS infrastruce like and how does it work with CF specifically? Any detailed examples you can share with me?

thanks.

Christopher

bctechnology@gmail.com

@Al,
this cfc doesn't work for me either because I have CF MX7. I just need to do 2 cfhttp requests. The first would set a session (page language). And the second should use the same session to obtain the response in the right language..
how did you solve your problem without this cfc?
thank you for helping!

@Joshua Shaffner,
after I did that I receive the error:
Context validation error for tag cfhttpparam.
The tag must be nested inside a cfhttp tag.

what else did you do? where did put the surrounding <cfhttp tag?

It's a fantastic and I have no troubles with it at all. :-) maintaining the session information across multiple CFHttp calls and gotten the secure page data was very useful for me. I tryed implementing it into our site www.scooble.de.

Somehow i got to your page but sorry i dont know something about programming or scripting. But one thing i can say - dude you look like Jack from Lost ;-)

CF 8 Standard Edition Issue

Hi Ben

Thanks for another excellent post. Trying to make this work on standard edition and I get this error:

"A License exception has occurred : You tried to access a restricted feature for the Standard edition: JSP
coldfusion.license.LicenseManager$LicenseIllegalAccessException: A License exception has occurred : You tried to access a restricted feature for the Standard edition: JSP"

The website I am trying to connect to might be using JSP pages I am not really sure but as far as I understand this is only an issue when the getPageContext() function is used. I can't see your cfc using this.

Any ideas on how this code can be modified to connect to JSP pages using Standard Edition.

Thank you

George

@George,

The server you are connecting to might be having licensing issues. The CFHTTPSession.cfc has no concept of target scripting languages, JSP, PHP, CF, ASP, etc. All it does it connect via HTTP calls. It's up to the target server to function properly.

Hey Ben,

Your site is quickly becoming my "go-to" place for CF answers. Great job!

I've bumped in to a similar "Connection Error" problem caused by server side compression that's been described above. My initial fix was to add the 'Accept-Encoding' parameters to each call before sending the request. This worked for some, but not all of the calls. Ask my wife, it's driven me crazy for 3 days.

The problem turned out to be redirects + server compression. It looks like HttpSession does not maintain header values from the initial request to the forwarded request. This means that the initial request goes through, but the page that you were forwarded to will still give you a "Connection Error". Damn.

My fix was to hack your original HttpSession object. I'm happy to share it with you if you'd like, but be warned: it no longer looks like the code you originally posted.

A simpler fix might be to force the Accept-Encoding value into every http call. It seems like enough people have had this problem that it would make sense to set this as a default value for every get() and post().

Thanks again for posting great insights, useful tools, and for digging in to the far reaches of ColdFusion.

@Ben,

Yeah, I've recently been hit with an accept-encoding issue when using the Campaign Monitor API (they turned on GZip). That's a good idea to put the encoding header in each request.

Just found a small bug in the CFHTTPSession component; Passing in a different UserAgent to the Init or SetUserAgent functions causes an exception.

This is caused because the SetUserAgent function has the wrong Argument name. (Value instead of UserAgent.) Easy enough to fix-- I guess nobody has used that feature up to now?

@Arthur,

Ah, awesome catch. I'll add that as a fix.

Today I came across teh same error mentioned here:

---
A License exception has occurred: You tried to access a restricted feature for the Standard edition: JSP
---

Can anybody tell me what IS the error? What restriction did I violate? This didn't happen on my CF7 but on CF9 Standard/Linux instead. :/

Ronald

@Ron,

I've seen that error happen when you try to Include() JSP pages. I think it was something like:

<cfset getPageContext().include( ...you jsp file... ) />

Of course, if you are getting this error hitting another page, it might be an error on *their* page and you just happen to see in your CFHTTP response?

@Arthur,

I finally got around to fixing this error. I updated the project page. As it turns out, the entire file-upload (file post) functionality was also completely busted - I guess no one ever tried to do that either.

hey that's awesome! we are finally upgrading from CF7 to CF9 so I will definitely get some great use out of this (instead of the ugly hacked up mess I ended up using) thanks & keep up the great work!

It worked for what I needed perfectly the first try... this is huge, you have made my week!

@Arthur, @Tom,

Thanks guys. Glad you're getting / going to get some value out of it. If you think of any upgrades, let me know.

Hi Ben
What can I say? whenever I type anything about coldfusion your site comes up, so sorry about hassling you!

I have a situation where I need to do multiple cfhttp requests in quick succession, as i'm getting feeds from lots of different urls is there a way that I can do these requests at once? I think waiting for each of these requests is slowing my page down (although it is getting cached every 20 seconds) The more feeds I add the slower the page will get and i'll end up with a huge lag time.

@Matt,

For something like that, I'd look into CFThread perhaps (if you are using CF8+).

For someone that more than frequently uses cfhttp, this, my friend, is the most bad ass-est thing I have ever seen! Words can't begin to describe how thankful I am to browse your blog...

@Jim,

Most excellent! I'm super glad you like it! I actually have an update to make to this project that I have failed to for sooo long. It has to do with the way some values get escaped in cookies. This reminds to actually do something about that :)

hi, thanks for this. really useful.

Just noticed your AddFile function is passing "Cookie" as the Type parameter when calling AddParam.

oops nevermind, i found your project page!

Post A Comment

Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.

Please review the following issues:

Author Name:
Author Email:
Author Website:
Comment:	Supported HTML tags for formatting: <strong>bold</strong>   <em>italic</em>   <code>code</code>
	Remember my information
	Subscribe to comments
	Send me a copy of this comment