The online and in-house securities trading systems of Phillip Securities in Singapore crashed on Tuesday, Jan. 15. The online system used by customers is Phillip's On-line Electronic Mart System (POEMS), the "Web site most-visited by Singapore users in [market research company Hitwise's] Business and Finance Stocks and Shares category", according to the company. Phillip Securities was an "eBroker of the Year" in 2000.
The in-house system used by the firm's traders also failed, so customers had no recourse to execute transactions from their Phillip Securities accounts while the systems were down.
Phillip Securities posted this apology on its home page: "An electrical power fault at our company's data center...resulted in a sudden power cut to all the hardware hosting our trading applications. We would like to express our sincere apologies for any inconvenience caused."
One reaction from "Goggle", presumably a customer: "They are lousy. I am seriously considering changing broking firm." The president of the Securities Investors Association of Singapore even publicly suggested a recovery plan to Phillip Securities customers: open accounts with Phillip's competitors.
I'll bet that's not the recovery strategy that Phillip Securities would choose.
I imagine the directors of parent company PhillipCapital will have plenty of opportunities to explain to clients--and to the Singapore Exchange--how they could have let this happen. Contingency planning is the firm's directors' responsibility, not the IT department's.
What should get the attention of planners making notes for their "Lessons Learned" files, however, were the questions asked by Chua Hian Hou from Singapore's English-language Straits Times newspaper:
• Don't you have a backup generator?
• How often have you had a power failure?
• What is your company doing to keep it from happening again?
When I role-play during a crisis exercise, I try to pose open-ended, embarrassing questions like those to a company's spokesperson to replicate the feeling of being under "live fire". The spokesman often gets a pained expression that conveys "only a rude ang moh would ask embarrassing questions like those". Well, that's just not true, is it?
What should the company's answers have been? I suppose that completely truthful answers might have been:
• "Well, we decided the cost of a backup generator was too high. Besides, Singapore has very reliable electrical power."
• "It's the property manager's job to maintain the power in our building, but we've never had a power failure in our data centre before."
• "With all this publicity, now we'll probably have to spend that money to buy a backup generator."
Clearly not what their PR advisor would recommend they say. If the company's managers and directors had rehearsed a good crisis communication plan, their answers could have been:
• "We invested in a backup generator last year so that our customers and brokers would never notice a power failure, which we didn't think was too likely anyway."
• "Very rarely, but serving our customers is our highest priority so we have developed disaster recovery, business continuity and crisis management plans."
• "We rehearse those plans at least twice a year, just in case."
Or something like that.
What's consuming your network bandwidth?
Find out today. Download SolarWinds award-wining software here - FREE 30-day Trial
Security Considerations for Cloud-Ready Data Centers - Download the whitepaper!
A network-centric approach to providing security in the data center delivers multiple benefits
Cloud Computing & Virtualization Conference 2009
Register & learn how to adopt cloud computing & virtualization technology to your company
IT Salary & Skills Report 2009
Join activeTechPros for free access to the report
There are currently no comments for this post.