Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at cf.Objective() 2014 (Bloomington, MN) with:

Reconciling Different Types Of APIs And What Data They Return

By Ben Nadel on
Tags: ColdFusion

As I have started to build larger, more intense ColdFusion and AJAX applications (powered by jQuery of course!), I have had trouble reconciling the intent of API decoupling and the usefulness of the data that gets returned. By this, I mean that in an ideal world, I'd like to return JSON data (or XML) in an API call such that the calling system (HTML, FLEX, AIR, etc.) and the Application are not tightly bound. But, at the same time, I do acknowledge that often times, it's a whole lot easier if the API makes some assumptions about the interface (ie. how fields are labeled).

For example, it's much easier to get back the error message:

"Please enter your username"

... than to get back some data exception structure like:

{ username: "required" }

... and then having to define a way to translate the "data error" into a "user interface error."

I am feeling very torn between these two worlds right now. I want to keep it clean, but I also need to accept that cleanliness is not an end in an of itself. I have, no joke, been struggling with this problem for many months. On the way to work, however, I think I had a moment of clarity. I tried moving the problem domain up a layer in abstraction and asked myself if the real problem was that I was actually trying to solve two different problems into the same solution?

So then I started to think maybe the real solution is to have two different sets of APIs: one that is publicly accessible and one that is internally accessible to the HTML client.


 
 
 

 
Different APIs And The Different Types Of Data That They Return.  
 
 
 

This is an architecture that I am playing around with in my head. In this scenario, the HTML client would have its own API, custom tailored to return User-Interface-friendly error messages (a collection of them). All other APIs, not tied to the HTML client, would return data-centric error collections that are completely UI-agnostic. Of course, if the HTML client were so inclined, it could call the public API as well, but then it would have the responsibility of displaying those in a coherent way. Of course, all APIs would be, internally, calling the Domain Model; but, they would need to handle all exceptions thrown by the Domain Model.

When I look at this architecture, my mind is put very much at ease. I don't keep thinking to myself - What happens if someone adds a new interface where it's called "Login", not "Username" - our error messages won't make any sense!! Finally, I can see that these would each be calling their own specialized API and the world seems right.

Of course, as my Domain Model trends more toward OOP (object oriented programming), I might find that I just want to roll the HTML API into the HTML Client. But, that's not a question I am prepared to think about just yet.

Anyone have any thoughts on this?




Reader Comments

I heartily agree with you that "cleanliness is not an end in an of itself". But what's so terrible about the client getting JSON or XML back from the server instead of user-friendly error messages?

Take any popular web service API as an example (Google, Yahoo, Twitter, etc.) When you call one of their API methods, you get a response that is generic and structured. Generic so thousands of different apps can all use the same API, and structured so that they can consistently pull out the information they need.

Not only does this approach let multiple clients use a service, but it lets clients and services change at different rates. You can tweak the UI every 2 days at the customer's request. The service API is free to evolve much more slowly, as they usually do.

This becomes infinitely harder when you couple the service API to a specific client implementation. And exposing multiple APIs for different types of clients leaves you way more complexity to manage. And complexity is a price you should only pay for looser coupling, not tighter coupling.

IMHO, an API should expect a structured request and return a structured response. There is a finite number of errors that can be returned, so you label them logically in the response. The client is then completely free to react in a zillion different ways to those errors.

I agree that, if it is clear that an API will only be used by a specific source/frontend, the hassle of first creating an XML or JSON structure and then evaluating this result and reformatting it into something readable is quite overkill and also crap from a performance standpoint (JavaScript receiving JSON data, parsing it and then having to apply other logic to it, for example). Also, if the structure behind the data is changed, you'd have to re-code the API AND the source receiving it.

Now, if that API will ever be used for something else, then a generic format will of course always be best.

I usually weigh out what's best for the current project in the long run. If it's a small site that will never have to talk to another source and just output a text error from an AJAX call, then that's what the API will send back. Call me a hack or lazy ;D

I agree that API's should return structured errors. I also agree that you are trying to solve two problems with one solution. I think you're on the right track, but it looks like your HTML API would do exactly the same thing as your Public API but return errors differently.

I'm not sure I would translate API errors to UI errors at all. I would argue that all data should be validated and any error messages displayed BEFORE passing it on to an API. If for whatever reason data passes validation and gets sent to an API which subsequently fails, the API should pass back something to help a developer figure out what's going on while the user just gets some generic "it failed" message. Some API's built by the big guys out there just pass back codes you need to look up online to figure out what's going on.

I think part of the issue is that the AJAX we write isn't always meant as part of an API but as Chris said, built for a specific page. I think a good analogy for a call to an actual API would be a cfquery call. Just like we don't use cfquery errors to determine UI problems, we shouldn't expect to use API errors for that either.

@David,

Maybe you are absolutely right. The issue I keep having is that it's a real pain in the butt to have the client take structured errors and turn them into meaningful errors (for the end-user). But, maybe that pain is me just NEVER sitting down and actually trying to come up with a way to perform that translation client-side.

Maybe, one structured response is really just the way to go about it.

I wonder how this would play with session management. Right now, I make my AJAX calls to CFM pages mostly (and some CFCs); but my API definitely assumes that session is handled automatically as part of the client request.

@Chris,

I am not sure there is really any performance issue with converting JSON to client-based data messages. I think it's more just an effort issue - I have not sat down and figured how to do this.

I am going to assume that it can be done easily - I just need to figure out how :)

@PJ,

Right now, I am going to assume that all data validation is done on the server-side via the API. I am comfortable with this so far and there are things that can only be checked via the server-side anyway. As I get more comfortable with the API-architecture, I can start to move things to the client as needed.

That being said, I think we are going to have core differences in what the API should do. Not to say that I'm not open to that... just that we are starting with different assumptions.

How would you do something like a Username availability check? Would you make a different API call just for that before submitting the form data to an API call? Curious how you handle things that can only be checked server-side in conjunction with things that can also be checked client-side.

Ben,
Have you ever taken a look at the Facebook API? I think that they do an awesome job with it. One of the things that you will notice is that you are actually friendly on their servers. They have you send all API calls as one API call.

What I mean is that instead of sending a call for username, a call for this, a call for that, everything is sent together right at the end of the request. I am not doing a great job of describing it, but you should really take a look at it.

I would also make all of your API calls return the same data, whether you are calling internally or externally. This takes out a lot of the effort for you. Maybe you could have an optional argument that lets the app know that it is an internal call (to avoid throttling or limiting internal calls).

Thanks,
Brandon Hansen

You make an interesting point! Even if I did check availability first, it's possible that somebody will register the name before I store can it.

If I wanted to validate everything server side, I would create a page that just validates form fields (check if their empty etc) and another to call an API that attempts registration. If the attempted registration failed, I would tell the user that the name is taken. So I guess the way I would go about it is to try and validate as much as I could before sending it. I wouldn't really call the method to validate the form fields part of an "API," just a different way to validate my form. The only method I would make part of a Data API is the method to attempt registration.

Just to clarify, to check if the fields are empty or of the proper length, I'd use JS to pass that data to a page that validates it. This page, since it's used specificly for a particular form and not part of a Data API can return the errors however I find most convenient.

ok, the performance argument was irrelevant, especially when javascript is dealing with its native JSON format.

what i do for simple error handling (let's continue with the form submission from your original post) is just pass back formatted HTML that then gets outputted in let's say a modal jquery ui dialag. i.e. <ul><li>Username required</li><li>Password required</li></ul> etc.
if a dialog that tells you which fields are missing is enough, that is it.

of course, if you pass back a JSON string such as { username: "required", password: "required" } you can then loop over those objects, highlight the input fields, write additional information next to those fields, etc., but you have to define all this logic both to your backend and your javascript code.

you could also just send back all the javascript statements in the ajax response and run them through an eval() and be done with it as well ;D

there are so many ways to tackle all the different situations, no wonder you are wondering about how to best optimize the API for specific frontends. i guess you should just go with works best for you for the task at hand and not worry about accepted API standards or whatever

Although your post is very "where do the messages come from" centric, one of the things I noted is that your "public" API is on the same "layer" as your private API.

One of the things that I am instituting at our work, and use in my own framework, is that the public facing API sits on a higher layer than my private layer, or service layer. With this setup our company offers a web services API for our customers to interact with their data in our system using the same business rules and transactions as our application, all by using the same service layer. Yes, the way the customer interacts with our API is through a seperate layer, and the calls are different, but they simply defer to the layer below to execute. It has worked wonderfully so far.

To see the chart visit http://www.adampresley.com/images/software-layer-architecture.png.

Cheers!

I like the terminology of private API (acting as sort of a service layer) and public API. At work we also have a public API that sits in a different layer. In calling our own pubic API, we usually have one AJAX call to the private API, which would in turn post to the public API to store the data. We validate data twice, once at each layer. The public API generates error codes used by developers for debugging and the private API generates error messages however we want, often in the form of UI friendly messages.

In the end, I think this this might have been what you were getting at with your HTML API.

@Brandon

The link has an extra fullstop at the end after the file extension (png.)

Just remove the fullstop and the link works fine.

@Adam,

I am not sure I follow what you mean exactly. From your diagram, it doesn't look like the View is calling the Service layer, but rather the Controller is calling the Service layer. Are you talking about AJAX-type calls, or server-side calls?

If you are talking about server-side, I would definitely agree with you - the Controller would be calling the Service layer directly. What I'm referring to, though, is calling the API from the View, not the Controller.

In both cases. My applications' view pages are handed data by the controller, and it in turn gets data from the service layer. AJAX calls in this particular case use the controller to execute calls. I do know that many frameworks prefer to put remote fascade objects on top of the service layer specifically for AJAX calls, and that's cool. In our app we define controller methods that transform results to JSON for AJAX calls, acting as a remote proxy.

@Adam,

I actually do very much the same thing in some of my apps - the AJAX is just a standard Controller call that returns JSON rather than HTML. I think it works great and really allows me to centralized all of my API calls. They all go through "api."... example:

api.contacts.quick-add
api.assets.upload
api.schools.get-schools

This way, all API try/catch and unified data returns are handled by my first "circuit" or whatever you want to call it. Works quite nicely.

Ben, I too have been struggling with very similar questions for a couple months. Even without AJAX involvement, I'm just questioning the way my domain objects are validated and how any errors are passed down to the view layer with friendly messages. I currently have a generic ValidationResult.cfc so I can consistently expect this object type from domain object validation throughout my app. It basically carries a success/fail property and a collection of errors. Each error item holds field, error type and error message. So, I have both a data-centric and view/user-centric error message in this collection.

To address the concern of changing UI implementations that you mentioned (your example of "Login" changing to "Username") I have a default error message for each property, but allowed for this to be easily overridden in an abstracted manner.

I'm not completely happy with this solution and I'm currently looking to abstract this one layer further. I was looking to use a separate validation definition, possibly in XML, which sounds extremely similar to Bob Silverberg's ValidateThis!, so I'll have to keep a close eye on that as well.

I should also mention that I still feel I'm on the n00b side of good OO design, so I may be way off base, but I'm trying :) As much as I've absorbed over the years, I still feel there's a tremendous amount of learning and experience to be gained still!

Cheers.

@Jamie,

I'm still very much on the nOOb site of OO also :) We're all learning here. I think Dave Stamm above gave me a good reality check - just return RAW data in the API and let the interface handle it.

My gut was reacting the idea that this would be / is hard. But, maybe that is because I have simply never sat down and though about how to easily translate data errors to UI errors.

If I could come up with something easy, I think that would make me feel the best about my API.

@Ben,

That does seem to "feel" right (returning raw data from the API). But could your initial idea of another layer still apply? Rather than being a separate API on the same level, maybe it's just an API facade, if you will, to handle the raw-to-interface-friendly translation. This may be a custom facade for each application/interface, or something reusable that simply has some configuration properties per application/interface. Make sense? Thanks!

@Jamie,

I was thinking that if there was something like a Form.CFC for each Form UI then perhaps the form could be submitted to that and it would handle returning friendly error collections. The Form.CFC instance would then, internally, take care of calling the service layer.

We'll see where this goes.

Ben,

I was calling facebook rest client API. to get the auth_token, but it gives me "Incorrect signature". Tried to find in google. but no help.

error_code 104
error_msg Incorrect signature
request_args
array
1
struct
key method
value facebook.auth.createToken
2
struct
key call_id
value 1XXXXXXXXXX
3
struct
key api_key
value 4aXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
4
struct
key v
value 1.0
5
struct
key signature
value c8XXXXX5XXXXXXX1da7XXXXXXXXXXX

@Nagarjuna,

I don't know very much about FaceBook development, but it sounds like you're not passing in the right arguments (or at least it cannot convert your arguments to the correct data types).