Financial Institution Reduces Capacity-related Outages by 20%

When you’re a Fortune Global 100 financial institution serving more than 52 million customers who interact with your brand every minute of the day, you can’t afford the slightest interruption to their day.

It’s simple. When the customer has a request, they want that request filled now.

Managing Blind

That was the biggest challenge facing Global Capacity Planning Team Leader at one of the world’s largest financial institutions. Before he arrived, “We were completely blind on the performance of our services,” he said. “And on the services where we weren’t completely blind, we had minimal information in a lot of cases. Our ability to monitor and understand our services and to plan adequately was greatly compromised.”

Managing the performance and capacity of 55,000 virtual and physical systems and supporting more than 10,000 applications, the old regime was unable to sufficiently address service performance issues.

“Change was needed,” he said. “Management realized the financial institution was exposed without a capacity management function.

“We’re in markets where every day is a working day to someone,” he said. “We have to provide the services they need and because our brand is in all the major markets, our services must be available 24 hours a day, 7 days a week, 365 days a year. We have zero windows where we can afford to have an outage.”

For organizations whose businesses are based on high level data transactions, like banks and stock exchanges, losses may be calculated in millions of dollars per minute of downtime, according to ITIC’s 2013-2014 Technology Trends and Deployment Survey.

“When stock market prices move 10 times every second, even one second means we could be out of sync with the rest of the market in terms of the prices we offer our customers,” he said.

For that reason, he deployed software that could collect performance data down to one-second granularity. The software also allowed him to take real data, drill the mathematical models, make projections, and accurately identify where bottlenecks would occur in the future.

“The business needs us to quickly answer if they need to factor in costs to provide new infrastructure to support their business strategies,” he said, “and they need to have confidence that the information we provide is correct.” Being confident in IT’s decisions is important. Banks are processing in excess of 100 million transactions per day from ATMs, stock trades, online banking, and other channels that could disrupt business environments.

The software provided the visibility the IT team needed, and they alerted business leaders that a planned upgrade would actually degrade performance. They were overruled by a technologist and the organization went ahead with the upgrade. Unfortunately, the outcome wasn’t good for business.

“What we predicted would happen, happened,” he said. “Our performance degraded, as we predicted, and our customers were unable to trade in the market for several hours while we rolled back the upgrade. Our ability to forecast what would really happen hastened our credibility with the business, which would’ve taken us months or even years to get.”

Black Friday Success

Black Friday is a great revenue opportunity for companies. The organization understood that its customers were going to shop virtually and physically, and they needed to provide high-quality service so their experience remained positive.

“We did some analysis based on what the markets - particularly the retail side - expected to see on Black Friday last year,” he said. “We identified where we had some potential issues with our Internet banking.”

After the expensive decision to overrule IT’s warnings in the earlier incident, the business took IT’s predictions to heart and made adjustments to pay for an increase in capacity to support the predicted traffic. And when Black Friday arrived…

“It was just seamless. The customers didn’t notice anything,” he said. “Black Friday was the busiest day we’ve ever had and the volume of calls to our call centers remained flat.”

Compliments to IT

His team has achieved a 20% reduction in incidents that stem from capacity-related root causes over the last two years. That is significant ROI.

“Our service availability has increased as a result of not shooting ourselves in our foot by running out of CPU and having to do upgrades in an unplanned manner,” he said.

The lack of outages are just one of many improvements he’s experienced since building his team. “We paid back our TeamQuest software purchase within a year.”

In addition, through the use of TeamQuest and the products embedded in the architecture, this financial leader has approval from lead regulators to record what’s going on and to model the company’s capacity management function based around the use of TeamQuest software.

“Now that isn’t what you’d call a traditional business success, but it’s viewed in that context here,” he said. “That means they’re happy and that we actually have proactive capacity planning in place.”

That means another happy customer in the always-on, 24/7/365 world that this business serves.