What's new in Socket.IO 1.0

I first stumbled upon Socket.IO when I was writing my book. The chapter was devoted to making a multiplayer version of a standalone HTML5 game. I built up a solid plan: describing transports, implementing ajax-longpoll, some notes about WebSockets (it was 2012, so WebSockets were not yet production-ready especially for mobile devices), and finally writing some logic on top of it. Then I saw Socket.IO and... why would I spend fifty pages writing about transports if I could spend five writing about Socket.IO instead?

This tool was simply great. The API was clean and intuitive and coding client-server calls felt natural. Of course it had its own problems. I remember spending an evening to debug it with my Samsung Galaxy v1 that was randomly breaking connections, but it worked at the end.

Then I forgot about it for some time. Couple of months ago I was conducting a "Node.js Bootcamp" training and Socket.IO was a part of my program. During the presentation one of my students told: "Socket.IO is dead, Guillermo is not updating it, just look at github page". And indeed, there was not to much activity in the github project.

Luckily, this was in no way the end of life for Socket.IO but rather the preparation for a major rebirth. Socket.IO 1.0 has landed just few weeks ago to show everyone that "the event emitter of the web" is alive and well.

This post is about the changes in Socket.IO 1.0 compared to the older versions. It covers both the changes in the architecture, npm module ecosystem as well as changes in API.

What's Socket.IO

Socket.IO is a JavaScript library that implements a real time communication between a browser and a server. It abstracts-out the details of underlying transport and provides some higher level abstractions like broadcasts, namespaces and rooms. Socket.IO is both a client and server library. The server by default is Node.js, however other third party ports are available. For example, you can run a Socket.IO server in Java and connect to it from the native ObjectiveC client.

The basic code for both client and a server is very straightforward and based on the idea of emitted events. You emit events on one side of the wire and get them on the other. As simple as that.

Client:

var socket = io();
  socket.emit('greeting', 'Hello, my name is Billy');
  socket.on('message', function(msg) {
  console.log(msg);
});

Server:

io.on('connection', function(socket){
  socket.on('greeting', function(text) {
      console.log(text);
      socket.emit('message', 'glad to see you');
  });
});

This code (omitting some trivial setup) is enough to start communication between a client and a server. The client initiates a connection and sends a greeting to the server. Server gets the greeting and replies with "glad to see you".

Note. If you did not write any Socket.IO apps before, a great place to start is the official tutorial that shows how to create a simple chat application - http://socket.io/get-started/chat/

New Modules

Before 1.0 Socket.IO was distributed as solid npm package that implemented every bit of functionality inside.

In Socket.IO 1.0 the code broke up into several small reusable modules less dependent on each other. The most notable shift is the extraction of low-level transport details into a separate project called Engine.IO. Now Socket.IO's job is to provide only higher-level facilities: multiplexing, reconnection, rooms, broadcasting, etc. All the heavy-lifting related to transports is done by Engine.IO.

This change is purely internal. If you are writing your code against Socket.IO you will not see the difference, as Engine.IO is not exposed "as is" to the client code. It is and important and very well-thought architectural change as it makes the overall project more maintainable. Besides, now there's a clear borders of components' responsibilities: Socket.IO is doing logic, Engine.IO is doing communication.

Socket.IO modules Ecosystem

Socket.IO and Engine.IO both are broken into core (socket.io and enigne.io), clients (socket.io-client and engine.io-client) and parsers (socket.io-parser and engine.io-parser).

The official part of the ecosystem looks like in the diagram below. The bold lines show actual NPM dependencies while thin black lines show logical relations. I omitted the repeating parts in the names of the projects, so -client in socket.io is socket.io-client and in engine.io it is engine.io-client.

This section describes modules behind each of ecosystems. We start from smaller and more independent modules and then move to higher-level abstractions.

  • engine.io - a transport engine. Abstracts-out the details of exchanging messages between a client and a server. It defines Socket - a class that hides the details of transports used by a client at a given time. This module low-level. For example there's no notion of types of events or rooms: only strings and binary arrays that travel between a client and a server. High level details are left for Socket.IO. You can use Engine.IO as a standalone server however in most cases it is better to use Socket.IO. This module uses engine.io-parser to encode and decode messages.

  • engine.io-client - a client implementation for engine.io that can run both in browser and Node.js. Uses engine.io-parser to encode/decode messages.

  • engine.io-parser - an implementation of engine.io protocol. Includes both encode and parse functions. Engine.IO uses a simple protocol to exchange binary or utf-8 messages. If there's no binary support on the platform, message is passed as Base64 string. The only meta-data sent along with the message is message type: ping, pong, data, etc. Parser can also work with batches: encode or decode multiple messages as one packet.

  • socket.io - that's what this post is all about. The high-level event transmitter that implements re-connects, rooms, broadcasts, multiplexing and other advanced features. It relies on engine.io for low-level transport, socket.io-parser for protocol handling, socket.io-client for client-side code and socket.io-adapter for tracking rooms and namespaces.

  • socket.io-adapter - having somewhat misleading name, an Adaptor is a registry of users, rooms and namespaces. It has only 4 methods:

    • add(): for adding a socket to a room
    • del(): to delete a socket from a room
    • delAll() - to delete a socket from all rooms
    • broadcast() - to send a message to all sockets in a room this class is a base for extensions. socket.io-redis builds a redis-based Adaptor that allows to distribute Socket.IO application between multiple nodes.
  • socket.io-client - a client for Socket.IO. You can use it from both browser and Node.js. Uses socket.io-parser to encode and decode messages

  • socket.io-parser - a parser and encoder of Socket.IO protocol, a reference implementation of socket.io-protocol. It is not an extension of engine.io protocol. It runs on top of engine.io just like REST is a protocol on top of HTTP and not an extension of HTTP. Socket.IO protocol describes the transformation from/to Engine.IO transportable entity plus Socket.IO specific fields like packet type or namespace for multiplexing.

  • socket.io-protocol - a description of Socket.IO protocol (not the implementation, this repository has only a readme.MD in it). This protocol is higher-level than the protocol of Engine.IO. For example, it supports sending different types of data: strings, JSON objects or binary data. Besides it supports matching requests and responses and marking messages for rooms.

  • socket.io-redis - a Redis-based implementation of an Adapter. With the help of this module you can set up a cluster of Socket.IO instances that will use Redis for communication between nodes.

  • socket.io-emitter - a Socket.IO-independent Node.js client that can send messages to redis-powered cluster

New Transport strategy

Before 1.0 Socket.IO was acting optimistic and first tried to establish the WebSocket connection. If that failed, Socket.IO downgraded to XHR. This approach could cause long delays if the environment was blocking WebSockets. Sometimes a client had to time out to know that WebSocket is not available.

Engine.IO implements a pessimistic but more user-friendly strategy. It starts from XHR longpoll and upgrades to WebSockets afterwards. Since XHR is available everywhere and less likely to cause problems, you get an immediate connection to the server and start to send and receive messages while engine.io is upgrading your connection. Over the last years this strategy has proven to bring better user experience most of the times.

New API

Socket.IO introduced several big changes to the architecture and API: middleware, finer-grained logging and client API changes.

Let's start with middleware as it is the most notable change. Just like in Express.js you can now register a function that is executed once the socket is created:

io.use(function(socket, next){
  count++;
  next();
});

This function can also perform more sophisticated logic like handling authorization, auto-joining some rooms or adding extra listeners for statistics. If you have some code in your Socket.IO application that does not implement core logic but rather handles an auxiliary aspect (authorization, logging, statistics, filters), it is a good candidate for becoming a middleware.

The other change is logging strategy. Socket.IO used to be quite verbose and it was lacking low-level control of individual components console output. Starting from version 1.0 Socket.IO is debug (https://www.npmjs.org/package/debug). Now it is a pleasure to configure logging. Just set a DEBUG environment variable and you get an output only from the components that you need. For example, this is the way to print only what a Socket object is having to say:

set DEBUG=socket.io:socket

There are more changes related to API, but since they are well listed in the official documentation, I will just leave a reference to the migration page: http://socket.io/docs/migrating-from-0-9/.

Summary

Summarizing the changes, Socket.IO 1.0 became cleaner, smarter and easier to extend than its 0.9x predecessors.

Links

Below are some links to github repositories that I referred in this post.

Comments

Please add your comments on github, or twitter @juriy.

Don't forget to follow, this will surely encourage more great posts!

Diving to Harmony: ECMAScript 6

ECMAScript 6 edition is the next big thing that is hiting the world of web development. ECMA-262, a standard behind JavaScript is about to be finalized which means that JavaScript as well as several other languages will soon be significantly improved. This post is a short introduction to a whole series of posts that cover ECMAScript 6 edition. This series is called "Diving to Harmony".

The last standard - ECMAScript 5 was published back in 2009. Last five years were quite intense for JavaScript and it became significantly more popular. Partly because of Node.js and serverside-js movement, partly because of HTML5 features that turned browsers into a powerful virtual machines capable of running applications never seen before.

We're writing way more JavaScript code than we used to write back in late 2000's. The more code you write the better you understand what is missing in the language. ECMAScript 6 is a natural evolution of a standard that is driven by community experience. It brings new features that will drastically change the way you think about JavaScript.

What's new

The change is big indeed. A previous version of the standard, ES5, mostly brought us new APIs, "strict mode", getters/setters, property configurations as well as several convenience features. The syntax hasn't changed and the language remained as we used to know it.

ES.next (a code name for 6 edition) is dramatically different from its predecessor. It is bringing new syntax, new keywords, new data structures and what's more important new language concepts. Below is a brief list of changes that we'll cover in the upcoming posts:

  • Block scoped let and constants
  • Default, spread and rest arguments
  • Arrow Functions
  • Iterators and generators
  • Typed arrays and typed objects
  • Destructuring and Array comprehension
  • Standard collections
  • Classes
  • Symbols
  • Modules
  • Proxy
  • Promises
  • Reflection
  • Template strings
  • Tail recursion

The standard itself grew from roughly 250 pages in 2009 to 600+ pages today. Many exciting things to learn!

Diving to Harmony

This post series brings together three aspects that will make you a better ES6 developer:

  • Simple and effective explanation of new language features with code examples (explaining what is a certain feature)
  • Best practices, use cases and pitfalls (explaining how ans and when to use it)
  • Implementation state in an existing JavaScript engines and libraries (explaining where to try the feature yourself)

Once you master what, when, where and how - you are ready to write a high quality ES6 code.

"Diving to Harmony" posts are linked with the text of a ECMA-262 version 6 draft. The links, text and examples are updated as the text of a standard updates (however do not expect too many changes as it is almost finalized).

Each post has a list of useful external resources that you might want to follow for extra details. Wherever possible I will point you to existing implementations of ES.next parts and encourage you to write some live code yourself. Developers are already building their frameworks around those yet-to-come features. Take a look at koa if you're curious.

Another resource about ES6?

There are other worthy resources around the net dedicated to ES6. The most notable are:

So why writing another post series? While other resources are definitely great (and I don't have a goal to compete in technical writing or skill with Nicolas or Dr. Rauschmayer) they mostly focus on listing the new features of a language acting as a friendly documentation enriched with code examples. "Diving to Harmony" shifts the focus towards emerging best practices. It is like a difference between knowing what Proxy can do and knowing how and when to use those features.

I hope that you will find this material useful. Since my blog does not yet have a descent commenting system (work in progress) feel free to send me an email to juriy.bura@gmail.com if you have thoughts to share. It is impossible to write a high quality material without your feedback. Besides, I'm always glad to hear from you!

Links:

Switching to static

I've got rid of Wordpress. During 8 years of blogging I was fighting with it. Everybody else around was using WP I thought that the problem was in me and all that pain is an essential part of tech writing. Wordpress was getting bigger and bigger and now it looks like an overly-complex engine for a small thing like a blog with several dozens of posts. It is still great for something more complex, but for serving several dozens of posts it is too much. For me personally the most useful feature of WP was commenting system but high volumes of spam drive standard Wordpress comments almost useless.

Docpad logo

The setup that I'm using right now is based on a great tool called Docpad (http://docpad.org). This generator of static pages takes a template and a bunch of posts and creates a static site that can be served by nginx, github pages, Heroku or any other service capable of hosting static content. Templates allow to insert just enough logic to do the tasks that I had so far. CSS and HTML is all hand-written by myself. Finally, I feel like I have control over my content.

What's best about this approach is that I can keep my posts in a in a writer-friendly markdown format and edit them any time. I hated to propagate changes from my drafts to online text editor that stores a post in a database. I hated to backup database each time I needed to save my posts. I hated to apply endless security patches. Now I have no database, my editor is Sublime Text and my site is as secure as nginx is.

To make setup even neater I’ve created a github repository for posts and added a webhook that calls a node.js webserver each time someone pushes new content. Node.js server in turn does git pull and re-generates the content of a blog. That’s quite close to a publishing system that I’ve dreamt of. The only thing that I’m missing is a way to edit and push new posts from my iPad, but I’ll figure out how to do that too.

Docpad is not the only tool in a family. In fact, there are many such tools. There's even a website that compares the most popular frameworks in each language and platform: http://staticgen.com/ so if you like the idea I recommend to take a brief look before choosing a tool that is right for you. In any case, if decide to change a tool later it should be relatively straightforward since all your posts and templates are independent from the generators.