Sep 30 2008 at 10:57am
How not to handle technical difficulties
This morning the CRTC (Canadian Radio-television and Telecommunications Commission) launched a “do not call” service. Within hours the website was not available. The explaination:
“A spokesperson for the CRTC said the site ‘worked fine’ when it launched at midnight, and said she didn’t know what had caused it to freeze up.
She speculated that the number of people trying to access the site may have blocked access to some users
‘Try it later and cross your fingers,’ she said.”
They don’t know what happened???? “Cross your fingers”?? Did they not realize that when they announced this they’d be getting a flood of traffic? Had their web developers never heard of the Digg effect? Similar problem when you’re announcing a very attractive service in media outlets across the country.
What do do when things go wrong
- Find out what actually happened.
- Fess up. Admit that your systems weren’t able to handle the traffic (or whatever the problem was). Do not blame the users.
- Promise to get the site working ASAP.
- Do get the site working ASAP.
- In the future, make sure your servers can handle the traffic, or plan other ways to avoid the problem.
Matt October 1st, 2008 at 9:33 pm
I kind of disagree with #5. It would have been nice if the systems were able to handle that kind of volume. But after the initial rush to register, I bet the demand for that is going to drop quite a bit. Why invest
thousands(oh, wait, this is the government I’m talking about) hundreds of thousands of dollars to support that initial burst of traffic? A do not call list doesn’t qualify as an vital service IMO.While I don’t know much about high traffic systems like this, I don’t think it’s just something you can switch on for a day or two and turn it off later. It’s probably a large, long term investment, which doesn’t really make sense for a once every three years service like the DNCL.
But hey, maybe it is a vital service for you guys. Telemarketers are pretty annoying.
Megan October 1st, 2008 at 10:22 pm
That is a valid point about the server investment in this case, although my points were meant to apply to any situation (not just a one-time launch). In this case Bell Canada was operating the system on behalf of the government and was able to increase capacity by the end of the day.
A better suggestion I saw on one of the news sites was to open the registration list in advance of the effective date. That way people could sign up gradually without a big flood all at once.
At the unviersity where I work we’re developing plans for emergencies, including serious tragedies such as the Virginia Tech shooting. We have to plan to keep our website up if something really bad happens and we’re on CNN. You have to plan for it somehow. The CRTC spokespeople seemed totally at a loss and couldn’t even properly explain the situation.
The other reason why I say that is because it’s just good customer relations. Remembers when that cuil search engine launched and they made a big fuss about it but then the site didn’t work? Will anyone use cuil again? In this case it’s the government, and you don’t expect too much, but it does add to widespread belief that the government can’t provide good service.
I agree that it’s not a vital service and we surely can live without it for a few days. On the other hand, what happened and the way it was handled made the CRTC look pretty dumb. Seems like they didn’t expect the media outlets to pick up on it and weren’t prepared for something like this to happen.
Liam October 2nd, 2008 at 12:10 am
As I understand the situation the Web site services are contracted out to Bell Canada. As a professional hosting company, that no doubt charges the government a fair wedge of cash, I’d say: yes, they can and should switch on more bandwidth and processing power.
There are many ways a professional hosting company can do this. If it’s a shared server (hosting lots of government sites), they could remove other sites on the same server. The database could be hosted on a seperate server, or multiple servers. Any non-essential images, stylesheets, and gratuitous Javascript could be removed during the times of high load. There are tonnes of things that could be done.
But no, they come out and say ‘try it later’. That doesn’t inspire confidence in the government, or Bell’s ability as a hosting provider!
Chuck Monroe October 19th, 2008 at 5:31 pm
Welcome back online
Megan October 19th, 2008 at 8:42 pm
Thanks, Chuck. I’ll post about what happened soon if I get a chance.
Stephen Bray January 2nd, 2009 at 2:36 am
What an amazing story. It just goes to show that using different media together can produce mega volumes of traffic.
I note that many people launching products on the Internet report ‘server-meltdown’. How much traffic can a dedicated server handle, do you know?
The tale is salutary, if you’re expecting a site-rush, unfortunately most of us aren’t, then you must make sure that your server can handle the load.
Stephen
http://www.stephenbray.com