We were approached by the Department of Health to further enhance their WordPress based website for the launch of the National Care Service. The launch was to have a live web video stream and The Department of Health wanted live text updates to accompany the video. The text updates would allow late comers to catch up, mean the launch team could provide annotations, links and accompanying materials as the launch progressed, and would provide a text record of the event afterwards. The projected maximum audience for this event reached well into the thousands, so the solution we provided would have to scale appropriately, while not consuming so many resources that the webserver could not fulfil it’s other duties (like serving other pages and so forth).
The structural foundation we use for the updates is WordPress comments, with some AJAX to fetch and display new comments every twenty seconds. When new comments arrive they are dropped in at the bottom of the update stream, highlighted, and the stream is scrolled to show them. This system works in any modern browser (and Internet Explorer), and even some mobile phones like Android, Palm Pre and Apple iPhones.
To test the system’s resilience we’ve worked with the site ISP, who provided a small array of servers allowing us to create a virtual tsunamis of requests, simulating the many thousands of users we were expecting. This exercise certainly proved the value of stress testing applications and not just assuming that what you have will work at scale; as demonstrated by the first technical solution we tried failing under the load. The problem we hit was that when more than 8,000 users are requesting updates every twenty seconds the database was crumbling under the pressure. Going back to the drawing board, we refactored our solution to completely bypass the database, and serve everyone from a single static file. The static approach has the benefit of avoiding both PHP and the database, and since Apache is extremely good at serving static files, it works considerably better (well over 300 times faster than when we were running the request directly to WordPress, and I think we could have scaled significantly higher if we’d had the need).
Once we had our solution in place and tested, the remainder of the work consisted of optimising the web server and WordPress to ensure we could handle a surge of traffic. We installed WP Super Cache, a stalwart caching plugin, along with some of our tried and tested tweaks to ensure optimum performance.
The event went off without a hitch, from our point of view (we’ll leave the political commentary to others), and we’re already planning enhancements to this system: What if the audience could vote? How about adding the capability to submit questions? The possibilities for this technology are very exciting… watch this space, and do get in touch if we can help you with a live event.
My colleague Simon at Puffbox has written up his thoughts on the event, and sheds some light on the strategy and history of the site.