Our web app at Tutorhub.com uses XMPP under the covers to send real time updates to in-browser clients. The results have been great, but we’ve found that as users sign up or modify their details, it requires a lot of configuration for the PubSub and PEP nodes. The web app sends configuration messages on the part of users, but the asynchronous nature of XMPP makes it harder to ensure that everything has worked smoothly and configuration succeeded.
We’ve identified possible solutions that involve custom ejabberd modules and the like, but this creates new problems: lock in, support difficulties (e.g. what happens when ejabberd internals get updated) and we lose features like decentralization.
Best practice in this space seems to be unknown and familiar web development concepts unused, so Theo has been toying with the idea of an HTTP gateway that operates in a similar way to RESTful services for operations where we care a lot that messages are delivered correctly, such as IQ stanzas and dataforms, and where efficient transport is less of an issue. The HTTP service would provide a convenient interface for configuring XMPP servers, probably using http basic access authentication to deal with login and then build upon the URI’s specified in the XEP’s to provide a very simple synchronous interface which other application developers can hook into.
We propose to put this out there under a BSD license, so if anyone has any input or ideas, please get in touch to the benefit of all.
P.S. While doing a bit of research, Theo’s come across a number of useful resources:
- http://idavoll.ik.nu/wiki/HTTP_Interface shows exactly what Theo was thinking of – but aimed solely at Pubsub (a damn good start), although it doesn’t seem to have been touched in 2 years.
- http://xmpp.org/extensions/xep-0060.html#registrar-querytypes gives out a URI structure that could be used as a well defined way of building the RESTful HTTP service
- http://xmpp.org/extensions/xep-0099.html provides a very consistent framework that would map very nicely onto RESTful HTTP service
- http://xmpp.org/extensions/xep-0147.html more information on URI’s
This week at Jiva we finished our ninth “sprint” of work on a project for Futurelab and we’re feeling proud and a little sentimental at seeing it released into the wild and taking our hands off the controls, for a while at least. The project known as Greater Expectations is funded by BECTA and is the brainchild of Bristol’s hi-tech education think-tank Futurelab. Their researchers and strategists conceived of what has, through our Agile development process come to be Infocow, a website listing online resources with the potential to empower young people.
Our development cycle in this project was one two-week iteration per month in order. This was designed to accommodate Futurelab’s rigorous programme of focus groups, user testing and consultation. Through this process, groups of young people from interested schools became ‘partners’ in Greater Expectations and shaped the features, design and identity of what developed into Infocow.
The project is now rolling out gently and there are a several more development iterations planned for the coming months. Looking back at the technical challenges, the interesting work fell into three broad areas:
Social networking tools
We knew from early research that social networks were likely to be key to Infocow’s success by viral marketing. We took the decision early to try integrating users’ Facebook networks into the site as a way to promote content demonstrated to be of interest to an individual’ peer group. Initially we worked with the Facebook API directly via the Facebooker gem (software library for Ruby, our language of choice). User feedback told us that what people really wanted was the ability to log in with their Facebook or WindowsLive credentials, so we backed out the Facebooker code and implemented the generalised authentication and social network interface from JanRain’s RPX. This allows users to login to Infocow using their existing social network account of choice and gives us a single API for dealing with these third parties. At present we can do everything we need with the RPX API, but we also have the option to fall back to lower level tools such as Travis Reeder’s mini_fb or the 3rd parties’ raw APIs if we need to get really creative.
Search and recommendations
Quite a number of mockups and wireframes were tossed around in the early days as we tried various interfaces for navigating the hundreds of websites listed. Following our Agile principle of trying out the simplest thing which could possibly work we went ahead and coded up a search interface based on Digg’s clean filtering system. This turned out to be super-easy using Mat Brown’s excellent Sunspot library which provides a Ruby wrapper for Solr the web component in Apache’s Lucene project, the Big Daddy of text search. As it’s early days and we have insufficient usage data to build a fully fledged recommendations systems, we’ve implemented a pretty naive recommender but look forward to plugging in some more sophisticated similarity and clustering algorithms.
Admin tools and reports
We created a whole suite of tools for administrators and ‘partners’ (youngsters recruited to moderate the site and curate content) to monitor and manage the websites listed and keep tabs on their popularity and comment-worthiness. A fair amount of work went into ensuring that both featured and user-generated content could be flagged as inappropriate by users or automatically picked up as being spam or obscene. Akismet, of course, is the defacto mechanism for testing content for spam, but we were surprised how short the standard lists of obscenities on the web seem to be, so we had to do a bit of collation of f*ck-filter lists to get to a point where some of our choice expletives were caught. If you have a particularly good list of swear-words, then please do email it in. We’ll put it to good use.
Whilst Jiva has to date been primarily a “pure-play” startup building applications such as Beanbag and Tutorhub, we have found this client project immensely rewarding. It has been a pleasure to work with the specialists at Futurelab who have a deep understanding of their field and to offer up our own imaginative technical solutions to fit the User Stories they provide. Hopefully, there will be plenty more to come.
Ed.
Should probably mention that http://tutorhub.com went into closed beta yesterday. Ping me if you want an invite.
Until now, we’ve not given much in the way of technical updates from Planet Jiva, but with the impending release of Tutorhub, I thought it would be worth sharing some of our technical trials and tribulations. From the word go, our technical vision required a need for real time ‘conversation’, blending IM style features with those found in run of the mill web applications. This is known as making life hard for yourself.
In contrast to the fundamentally asymmetric, request-response model that underlies almost all web based communication (even Web 2.0 sites that provide dynamic updates of information), instant messaging applications require any connected party to be able to send or receive a message at any time. Not all messages receive, or require, a response. The established open standard, used by Jabber, Google Chat and Facebook Chat, is XMPP. The recent release of the draft standard for the BOSH protocol, together with the emergence of a couple of open source JavaScript libraries which support XMPP over BOSH, allowed us to embed XMPP based communication channels within Tutorhub, our new web application for the online tutoring market. It’s been a complex undertaking. It necessitates moving from a two party request-response communication pattern to an architecture where the web browser must communicate both with the web server (via HTTP) and with the IM server (via XMPP over BOSH). In addition the IM server and the web server must route messages to one another, mediated by software ‘bots’. These messages must be queued to ensure that they are reliably received and processed asynchronously without causing undue delay in the request-response communication. We used AMQP for this, another emerging open standard developed for use in high volume transaction processing environments such as banking. XMPP allows us to effectively route queries to relevant tutors based on the subject of the query, without requiring tutors to constantly poll the server for updates, a practice known as ’semantic query routing’. The result? Hopefully, a much more scalable architecture.