Ben Nadel
On User Experience (UX) Design, JavaScript, ColdFusion, Node.js, Life, and Love.
I am the chief technical officer at InVision App, Inc - a prototyping and collaboration platform for designers, built by designers. I also rock out in JavaScript and ColdFusion 24x7.
Meanwhile on Twitter
Loading latest tweet...
Ben Nadel at NCDevCon 2011 (Raleigh, NC) with:

Realtime Messaging And Synchronization With NowJS And Node.js

By Ben Nadel on

Realtime communication is definitely the future of web application development. Not only does it make for a better, more natural user experience, it also simplifies the development process quite a bit. Gone (or going) are the days of polling - we are about to enter the era of bi-directional communication where "Push" is King. In the past, I've explored services like Pusher for ColdFusion-based realtime communication. Today, I started playing around with NowJS - a realtime messaging services that adds a bit of "magic" to your Node.js application.

 
 
 
 
 
 
 
 
 
 

I would say that NowJS is adding realtime messaging capabilities to your Node.js application; but really, I don't think that would be doing it justice. Yes, with NowJS, you can "push" notifications to the client directly from the server; but, the real magic here is that this feature is made possible by the fact that NowJS, quite literally, blurs the line between server and client.

This fuzzy overlapping of local and remote contexts is built around the "now" scope. On the client-side, this now scope builds a one-to-one relationship. On the server-side, this now scope builds both a one-to-one and a many-to-one relationship. That is, the client can only talk to the one server; but, the server has the ability to push messages to all clients, just one client, or any combination in between.

 
 
 
 
 
 
NowJS creates a magic pocket that is shared between the Server and the Client for amazing, realtime communication. 
 
 
 

And, to make this "worm hole" of a service even more mind-boggling, it allows client-based functions to be invoked on the server; and, it also allows server-based functions to be invoked on the client. I could try to describe this further but, I think this is the sort of thing you just have to see to believe.

To demonstrate the realtime, bi-directional communication provided by NowJS, I have built a very small Node.js application. In this application, the user can click on an image and drag it around the screen. The position of the image, as it moves about the screen, is broadcast from the "master" client up to the Node.js application, where is subsequently pushed down to all "slave" clients. These slave clients then receive the broadcast and update the position of their local image instances.

First, let's take a look at the Javascript file that defines our Node.js server-side application. When reading the following code, you can pretty much skip the first half - it just sets up the core HTTP server; the really interesting stuff is in the second half where the NowJS module gets initialized.

Server.js

  • // Include the necessary modules.
  • var sys = require( "sys" );
  • var http = require( "http" );
  • var url = require( "url" );
  • var path = require( "path" );
  • var fileSystem = require( "fs" );
  •  
  •  
  • // ---------------------------------------------------------- //
  • // ---------------------------------------------------------- //
  •  
  •  
  • // Create an instance of the HTTP server.
  • var server = http.createServer(
  • function( request, response ){
  •  
  • // Get the requested "script_name". This is the part of the
  • // path after the server_name.
  • var scriptName = request.url;
  •  
  • // Convert the script name (expand-path) to a physical file
  • // on the local file system.
  • var requestdFilePath = path.join( process.cwd(), scriptName );
  •  
  • // Read in the requested file. Remember, since all File I/O
  • // (input and output) is asynchronous in Node.js, we need to
  • // ask for the file to be read and then provide a callback
  • // for when that file data is available.
  • //
  • // NOTE: You can check to see if the file exists *before* you
  • // try to read it; but for our demo purposes, I don't see an
  • // immediate benefit since the readFile() method provides an
  • // error object.
  • fileSystem.readFile(
  • requestdFilePath,
  • "binary",
  • function( error, fileBinary ){
  •  
  • // Check to see if there was a problem reading the
  • // file. If so, we'll **assume** it is a 404 error.
  • if (error){
  •  
  • // Send the file not found header.
  • response.writeHead( 404 );
  •  
  • // Close the response.
  • response.end();
  •  
  • // Return out of this guard statement.
  • return;
  •  
  • }
  •  
  • // If we made it this far then the file was read in
  • // without a problem. Set a 200 status response.
  • response.writeHead( 200 );
  •  
  • // Serve up the file binary data. When doing this, we
  • // have to set the encoding as binary (it defaults to
  • // UTF-8).
  • response.write( fileBinary, "binary" );
  •  
  • // End the response.
  • response.end();
  •  
  • }
  • );
  •  
  • }
  • );
  •  
  • // Point the server to listen to the given port for incoming
  • // requests.
  • server.listen( 8080 );
  •  
  •  
  • // ---------------------------------------------------------- //
  • // ---------------------------------------------------------- //
  •  
  •  
  • // Create a local memory space for further now-configuration.
  • (function(){
  •  
  • // Now that we have our HTTP server initialized, let's configure
  • // our NowJS connector.
  • var nowjs = require( "now" );
  •  
  •  
  • // After we have set up our HTTP server to serve up "Static"
  • // files, we pass it off to the NowJS connector to have it
  • // augment the server object. This will prepare it to serve up
  • // the NowJS client module (including the appropriate port
  • // number and server name) and basically wire everything together
  • // for us.
  • //
  • // Everyone contains an object called "now" (ie. everyone.now) -
  • // this allows variables and functions to be shared between the
  • // server and the client.
  • var everyone = nowjs.initialize( server );
  •  
  •  
  • // Create primary key to keep track of all the clients that
  • // connect. Each one will be assigned a unique ID.
  • var primaryKey = 0;
  •  
  •  
  • // When a client has connected, assign it a UUID. In the
  • // context of this callback, "this" refers to the specific client
  • // that is communicating with the server.
  • //
  • // NOTE: This "uuid" value is NOT synced to the client; however,
  • // when the client connects to the server, this UUID will be
  • // available in the calling context.
  • everyone.connected(
  • function(){
  • this.now.uuid = ++primaryKey;
  • }
  • );
  •  
  •  
  • // Add a broadcast function to *every* client that they can call
  • // when they want to sync the position of the draggable target.
  • // In the context of this callback, "this" refers to the
  • // specific client that is communicating with the server.
  • everyone.now.syncPosition = function( position ){
  •  
  • // Now that we have the new position, we want to broadcast
  • // this back to every client except the one that sent it in
  • // the first place! As such, we want to perform a server-side
  • // filtering of the clients. To do this, we will use a filter
  • // method which filters on the UUID we assigned at connection
  • // time.
  • everyone.now.filterUpdateBroadcast( this.now.uuid, position );
  •  
  • };
  •  
  •  
  • // We want the "update" messages to go to every client except
  • // the one that announced it (as it is taking care of that on
  • // its own site). As such, we need a way to filter our update
  • // broadcasts. By defining this filter method on the server, it
  • // allows us to cut down on some server-client communication.
  • everyone.now.filterUpdateBroadcast = function( masterUUID, position ){
  •  
  • // Make sure this client is NOT the same client as the one
  • // that sent the original position broadcast.
  • if (this.now.uuid == masterUUID){
  •  
  • // Return out of guard statement - we don't want to
  • // send an update message back to the sender.
  • return;
  •  
  • }
  •  
  • // If we've made it this far, then this client is a slave
  • // client, not a master client.
  • everyone.now.updatePosition( position );
  •  
  • };
  •  
  • })();
  •  
  •  
  • // ---------------------------------------------------------- //
  • // ---------------------------------------------------------- //
  •  
  •  
  • // Write debugging information to the console to indicate that
  • // the server has been configured and is up and running.
  • sys.puts( "Server is running on 8080" );

Once the core HTTP server is configured and the NowJS module is initialized, we are given access to the "everyone" object. This everyone object then provides us with access to the server-side "now" scope. This "now" scope is shared between the server and every one of the clients. Anything added to or removed from the server-side "now" scope is also added to or removed from every client currently (or eventually) connected to the server.

This is true for both variables and functions! Notice that my server-side Node.js code defines two methods: syncPosition() and filterUpdateBroadcast(). By defining them in the "everyone.now" scope, I am making them available to both the server and to every single one of the connected clients.

But what about that, "everyone.now.updatePosition()", function? Where did that come from? Ah-ha! Here's the real, "there is no spoon" mind-screw - that function is defined on the client (which we'll see in a minute). And, since it's defined in the client's "now" scope, the server-side Javascript can then invoke it as if there were no separation between the server and client contexts.

Now, let's take a look at the client-side code. The bulk of this code has to do with the setting up the draggable-image functionality; the cool parts all take place in the client-side "now" scope.

Index.htm

  • <!DOCTYPE html>
  • <html>
  • <head>
  • <title>NowJS And Node.js Realtime Communication</title>
  •  
  • <style type="text/css">
  •  
  • html,
  • body {
  • height: 100% ;
  • overflow: hidden ;
  • width: 100% ;
  • }
  •  
  • img {
  • left: 9px ;
  • position: absolute ;
  • top: 70px ;
  • }
  •  
  • </style>
  •  
  • <!-- We have this file stored explicitly. -->
  • <script type="text/javascript" src="./jquery-1.5.2.js"/></script>
  •  
  • <!--
  • The NowJS HTTP augmentation will take care of routing
  • this - we don't actually have this physical file stored
  • at this file path.
  • -->
  • <script type="text/javascript" src="/nowjs/now.js"></script>
  • </head>
  • <body>
  •  
  • <h1>
  • NowJS And Node.js Realtime Communication
  • </h1>
  •  
  • <!--
  • This will be draggable. When this image drags, we are
  • going to sync the position of it across browsers.
  • -->
  • <img
  • id="olivia"
  • src="./olivia_williams.jpg"
  • width="100"
  • height="100"
  • alt="The very gorgeous Olivia Williams."
  • />
  •  
  •  
  • <!-- Configure the client-side script. -->
  • <script type="text/javascript">
  •  
  • // Get a reference to the target draggable.
  • var olivia = $( "#olivia" );
  •  
  • // Get a reference to the body - this is the element on which
  • // we'll be tracking mouse movement once the draggable
  • // tracking has been turned on.
  • var body = $( "body" );
  •  
  •  
  • // On mouse-down, turn on draggability.
  • olivia.mousedown(
  • function( event ){
  • // Prevent the default behavior.
  • event.preventDefault();
  •  
  • // Get the current position of the mouse within the
  • // bounds of the target.
  • var localOffset = {
  • x: (event.pageX - olivia.position().left),
  • y: (event.pageY - olivia.position().top)
  • };
  •  
  • // Start tracking the mouse movement on the body.
  • // We're tracking on the body so that the mouse can
  • // move faster than the tracking.
  • body.mousemove(
  • function( event ){
  • // Create a new position object.
  • var newPosition = {
  • left: (event.pageX - localOffset.x),
  • top: (event.pageY - localOffset.y)
  • };
  •  
  • // Update the target position locally.
  • olivia.css( newPosition );
  •  
  • // Announce the updated position so that we
  • // can sync accross all clients with NowJS.
  • now.syncPosition( newPosition );
  • }
  • );
  • }
  • );
  •  
  •  
  • // On mouse-up, turn off draggability.
  • olivia.mouseup(
  • function( event ){
  • // Unbind the mousemove - no need to track movement
  • // once the mouse has been lifted.
  • body.unbind( "mousemove" );
  • }
  • );
  •  
  •  
  • // I allow the remove server to make a request to update the
  • // position of the target.
  • //
  • // NOTE: By defining this function in the NOW scope, it gives
  • // the server access to it as well.
  • now.updatePosition = function( newPosition ){
  •  
  • // Check to see if this client is in master mode; if so,
  • // we won't update the position as this client is
  • // actively updating its own position.
  • olivia.css( newPosition );
  •  
  • };
  •  
  • </script>
  •  
  • </body>
  • </html>

When the user moves the image on the client, the client broadcasts the new position using the "now.syncPosition()" function. This function, which was defined on the server, then pushes the updated position down to all the other clients using the "now.updatePosition()" function, which was defined on the client.

Even after coding this myself, it's still somewhat confusing; so, let's look at a quick rundown of the various functions to see where they were defined and where they were invoked:

  • syncPosition()
    • Defined: Server-side
    • Invoked: Client-side
  • filterUpdateBroadcast()
    • Defined: Server-side
    • Invoked: Server-side
  • updatePosition()
    • Defined: Client-side
    • Invoked: Server-side

Is your mind sufficiently blown yet?

Why Use The Server-Side Function filterUpdateBroadcast()

Good question! It's there to make sure that the "master" client doesn't receive its own broadcasts. But, this part of the code is so magical that I don't even fully understand how it works. As I said above, defining a function in the server-side "everyone.now" scope makes it available to each client. However, according to the "Best Practices" page on NowJS, using a function like this allows for server-side-only filtering that supposedly prevents unnecessary messages from being sent to the client.

But, if the "everyone.now" scope is available on the client, how are we preventing server-client communication? I simply don't know. If I had to hazard a guess, I would say that NowJS is maintaining a mirrored copy of every client scope - a mirrored copy that is "eventually synchronized" with the client. But honest, I can't even imagine how this stuff is technically enabled.

Of course, that's the beauty of a blackbox service - you don't have to understand it - it just works!

Realtime communication is going to become a mainstay of web application development. It's nice to see that libraries like NowJS are making it so easy to implement in a Node.js environment. I know you're probably still confused after this blog post (I know I am); but, hopefully this has at least got you wanting to know more.




Reader Comments

Note that your example has security vulnerabilities.

For example, I could mimic to be the master for everyone rather than a slave by changing now.uuid.

While for your image example this is not a problem, developers must be aware of this issue.

To solve it, refrain from syncing variables and initialize NowJS with clientWrite: false. You can then create methods such as setUserId(newId) which is executed on the server side, allowing you to validate this change and block any malicious requests.

Reply to this Comment

@Tom,

Oh cool - I didn't know about the "clientWrite:false" initialization option. I'll have to look into that. I didn't see it in the guide anywhere; perhaps time to dig into the actual source code :D

@Phillip,

Pretty awesome, right?!

Reply to this Comment

Epic! I was just talking about needing a solution like this too.

It reminds me of the old days when we coded server-side ActionScript for Flash Communication Server [now FMS].

Reply to this Comment

Can you explain how this is better than Cirrus or Stratus and how it is better than MMO MMORPG servers - including things like wowza ?

Reply to this Comment

Nice but it is funny to see people going crazy about that when we have been able to do that for many years with XMPP. Anyway, I am glad people start to like the real-time web!

Reply to this Comment

Adrian not everyone knows of XMPP or care to setup an xmpp server. An "all js" solution is highly appealing.

As for my excitement, this is a highly simplified way to do advanced programming. It brings up all the possibilities of simplicity w/ FMS 9-10 years ago in an easier package.

Reply to this Comment

John, I totally agree with you. I guess we have failed to make it simple enough so that the community would just jump on it.
Anyway, I do believe that those technologies can live together as a symbiosis :)

Have a nice weekend!

Reply to this Comment

Yep. Folks jump for joy at a powerful, stable, "complete" few lines of code.

I agree, they def' can live together.

You too. Thanks! [re: have a nice weekend]

Reply to this Comment

@Aristophrenia,

I am not familiar with those other things. Perhaps I can answer if you are more specific with your question - how is this better, in what way exactly?

@Adrian,

To be honest, I am not even sure what XMPP is. I believe I have heard of it; but I would be hard-pressed to define it. I think a lot of technologies are simply defined by being in the right place at the right time and the right level of difficulty.

I just happened to be playing with Node.js and in a previous post someone just happened to mention NowJS and I just happened to be able to set it up :) Just a lot of good timing. I've used and heard of other Realtime messaging systems and, as @John was saying, these can all exist in the same world. After all, I'm primarily a ColdFusion developer so if there was a super easy way to implement Realtime in CF, I'd jump all over it.

@Tavs,

Glad you like!

Reply to this Comment

I'm not sure I full understand what is happening with node.js perhaps -maybe I am missing something entirely ? Hope you can explain it.

Wowserver (there are heaps of these types of servers), are servers which run all the massively multiplayer role playing games - they provide thousands of simultaneous connections relatively simply.

Cirrus is the next generation of Stratus - this is a genuine Peer to Peer communication system in flash. The flash players communicate directly with each other. No need for any serverside stuff at all. Once the connection is made thats it. It is also peer to cloud.

Can you tell me what the advantages are over these alternatives, or perhaps how they differ ?

Reply to this Comment

  • var requestdFilePath = path.join( process.cwd(), scriptName );

This looks like it will give a client access to any file on the server that the server process has read access to. E.g:

http://your.server.example/../../../etc/passwd

Reply to this Comment

@Aristophrenia keep in mind that it uses js - no need to install flash, js is supported across most of devices nowadays

i was a hardcore pro-flash, developing since flash mx was around and, sadly, i'm moving further and further away from it ( it proves useful only on limited scope now, just my opinion )

and with this js-trickery it get's a lot easier to develop realtime-communication services ( possibilities are endless )

@ben nice post, i'll keep on reading those articles - fun way to dig deeper

---

looks like it's time to dig into node.js :-)

Reply to this Comment

@Aristophrenia,

Node.js is written in Javascript, so if you know Javascript, there is going to be less of a learning curve, perhaps. If there are other severs that allow for large, parallel systems, then I couldn't say in which ways specifically one might be better than the other without being versed in each of them.

As for peer-to-peer, this wouldn't be a Node.js consideration as peer-to-peer doesn't involve a server (at least not for the primary actions). As such, there would be no reason to compare Node.js to anything that is peer-to-peer.

Really, however, I don't know enough on any of these technologies to comment.

@Martin,

Ah, good point! I am not sure how that kind of security concern is dealt with. There are web frameworks for Node.js at this point (which I have yet to look into). I assume they examine the incoming requests for ".." traversal requests and simply return 500 or 400 errors or something.

Thanks for pointing that out, however - it had not even crossed my mind!

@Stryju,

Thanks for weighing in on the matter - I was way out of my league to answer @Aristophrenia's questions.

Reply to this Comment

@Markval,

I *believe* that most of the "real-time" services attempt to use the WebSocket connections by default; and then, if they can't, they fall back to using things like polling and Flash Web Sockets. But, that is just a guess.

Reply to this Comment

There must be a problem on server.js code:

  • everyone.now.updatePosition( position );

should be:

  • this.now.updatePosition( position );

In case of everyone.now.updatePosition( ), It happens more function calls.

Reply to this Comment

Ben, thanks for the nice article. A.J, thanks for the correction, i tried to do change from "everyone" to "this" and it solved a skiping problem at the master client i was experiencing.

Reply to this Comment

Sorry for my unclear post. Skiping = skipping, meaning that when dragging the image around, the call to now.syncPosition(newPosition) made the browser kinda chop the movement of the image so it seemed to move in steps and not smooth.

Thanks again

Reply to this Comment

@A.J,

Ah, nice call. I believe you are right; I don't have the code in front of me (to test), but that would make more sense (since we are already in the context of an everyone function). Thanks!!

Reply to this Comment

I can't get this blog post out of my mind. This is the future (present day?) of the web!

I need more node.js! Where can I get started?

Reply to this Comment

@Phillip

Really, the Holy Grail? The future of the web? It can't be the future because people have been doing it for a decade. It's also not the best or most feature rich way of doing this.

Reply to this Comment

@Papichulo,

Ooo, Ooo, Ooo. Tell me more.
No seriously, tell me more.
Right now I've got a card game where each user watches a timer go down to 0 and then their page refreshes.

I'd like to rework it such that it's more real time.
OK, OK, since you asked: I teach a class and one of the chapters is in "the history of technology". It's 35 pages of names and dates...

So I wrote a card game where you play a card ("The IBM PC is released") and your opponent plays a card ("A hydrogen bomb is tested in the atmosphere") and whoever's card is the higher wins 2 points.

Reply to this Comment

Hello!
In my application for synchronization I use Socket.io's websockets and it's io.sockets.emit() method for broadcasting. Which method is used for synchronizing browsers?

And second question: what will happen if two users if two users simultaneously drag the picture?

Reply to this Comment

Thanks Guilherme!

I tried logging into the same session on two different Chrome screens and the second screen never got past the initialization.

Reply to this Comment

@Phillip,

I've restarted the app... I just keep creating 'rooms' but never remove them, so we might run into performance issues (FYI it is hosted on Nodester).

It should work now (couldn't really test from my end - going through ISP issues).

It worths reminding that as NowJS uses Socket.io under the hood. So it falls back to other methods (flash socket, json polling...) if your browser doesn't support or have WS enabled. But, for a better performance, you should be running these apps on a Websocket enabled browser (e.g. Chrome).

If it doesn't, please take a look at it on the GitHub link I sent. You can easily run it local.

Thanks for trying it out. And please let me know if I can help you some how.

Reply to this Comment

It never went past the "joining room" for the 2nd chrome window.
That's ok - I've got plenty of other technology I've got to learn without adding node to the list.

Reply to this Comment

This example looks great! sadly i can test it because i get this error all time:
[root@mail v2]# node hello.js

node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Cannot find module 'now'
at Function._resolveFilename (module.js:332:11)
at Function._load (module.js:279:25)
at Module.require (module.js:354:17)
at require (module.js:370:17)
at /var/www/vhosts/000000.com/v2/hello.js:86:13
at Object.anonymous(/var/www/vhosts/0000000.com/v2/hello.js:161:2)
at Module._compile (module.js:441:26)
at Object..js (module.js:459:10)
at Module.load (module.js:348:31)
at Function._load (module.js:308:12)

Reply to this Comment

In socket.io we have something like " var socket = io.connect('http://10.221.48.133:8080'); " , So that we can connect the client internally in the application. Isn't there such functionality in nowJS?
I'm launching my app directly, not through server( i mean not through something like http://localhost:8080/index.html). And I want the app to be connected to server internally using something like var now = io.connect('http://10.221.48.133:8080');
Is it possible?
Thanks in advance :)

Reply to this Comment

@taylerdurden,

I don't know if you have found what your looking for, but I think you are looking for something like

nowjs.on("connect", function(){
console.log("Joined: " + this.now.name);
});

nowjs.on("disconnect", function(){
console.log("Left: " + this.now.name);
});

Reply to this Comment

Post A Comment

You — Get Out Of My Dreams, Get Into My Comments
Live in the Now
Oops!
Comment Etiquette: Please do not post spam. Please keep the comments on-topic. Please do not post unrelated questions or large chunks of code. And, above all, please be nice to each other - we're trying to have a good conversation here.