Telecom Firm Eliminates Bottlenecks and Guarantees High Availability

Summary 

A couple of summers back, this telecom service provider suffered slowdowns on a key service it delivered to customers. On the recommendation of one of its largest clients, it obtained Vityl Capacity Management on a trial basis to investigate a bottleneck on its Oracle database server. Using the software, they were immediately able to zero-in on the cause — a lack of I/O paths to disk. They addressed the constraint, and consistent service delivery was restored. As a result of that successful trial, the company adopted Vityl Capacity Management and now utilizes it in many other areas of the enterprise. 

Buy More Hardware 

The basic technology architecture at this leading global communications service provider is a web layer, an application layer and an Oracle database layer. SunFire servers are utilized in a clustered environment, and multiple applications reside on each server. Load balancing and redundancy is employed to optimize performance and minimize downtime. More recently, the company has introduced Dell commodity boxes in a big way, as well as Red Hat Linux OS. VMware is currently in the test and development phase. 

In the past, the company primarily relied on the “Buy More Hardware” philosophy as a solution to all problems, and any capacity planning was done manually using spreadsheets. “We were in ‘react’ mode and had no time for analysis,” said the IT manager. “Capacity growth was based on estimates of new customer volume and a general approach of over-engineering to keep the pager from going off.” 

Whenever a problem cropped up, he had to go to application experts to find out what might be happening. This, however, led to many different opinions. 

When new applications came online, growth rates over the next six or twelve months were largely unknown. By overbuilding, the company paid heavily for unutilized resources (CPU, memory or disk), not to mention all of the other associated costs. Further, many application support teams had been developing performance interfaces without standardization. What initially led the organization to Vityl Capacity Management was an issue with an application in the summer of 2007. 

“We observed some capacity issues on an Oracle database server,” said the IT manager. “We had a large number of records to process and the database couldn’t keep up with the demand.” 

The company services well-known names in the telecommunications business. One of the largest global players, and a key customer, recommended Vityl Capacity Management to help analyze ongoing performance issues during a customer review which was later described as a humbling experience due to the severity of the outage in an application. 

“This telecommunication giant had integrated Vityl Capacity Management into its IT culture,” said the IT manager. “That indicated to me right away that there had to be something there that we could gain value from.” 

He immediately obtained a demo copy of Vityl Capacity Management. But instead of using it in a small-scale test environment, he put it to work to address the problems. He did a proof of concept of Vityl Capacity Management on the application and it pinpointed an IO issue. Based on the results of that trial in a real-world situation, the company decided to move forward and buy Vityl Capacity Management. 

The organization initially used Vityl Capacity Management to model workloads connecting with this application and its Oracle database. As soon as the IO bottleneck was discovered, the solution became obvious. Modeling revealed that increasing the number of paths to disk would alleviate any IO constraints. Accordingly, the company made the necessary disk allocation improvements, RAID upgrades and path additions to help bring about the processing of far more transactions. 

By providing four paths to the SAN over 4 Gig connections, adding a more powerful Sun server for the database and additional disk-based storage, all slowdowns were eliminated. While that effectively ended any concerns with that application, IT continued to use Vityl Capacity Management to monitor this mission-critical aspect of its service portfolio. 

“By staying on top of that application for another ten months, Vityl Capacity Management told us that we had a developing CPU capacity issue,” said the IT manager. “That led us in the direction of making a substantial CPU upgrade on our Oracle database server.”  

“Vityl Capacity Management has become an invaluable addition to our toolset. It makes it easy to anticipate bottlenecks and cope with rapid growth by allowing us to determine the necessary improvements to our systems.” 

Broad Vityl Capacity Management Use 

Since purchasing and installing Vityl Capacity Management, the company has applied it to 60 production servers and uses it across multiple applications. This includes a vital reporting engine used throughout the enterprise. The power of Vityl Capacity Management is harnessed continually on this application to ensure SLAs are fully maintained and exceeded. Vityl is also extensively used on a billing application as well as another key application. 

“Vityl Capacity Management has become an invaluable addition to our toolset,” said the IT manager. “It makes it easy to anticipate bottlenecks and cope with rapid growth by allowing us to determine the necessary improvements to our systems.” 

The company has since integrated more of Vityl Capacity Management into its operations. Vityl is used to display daily and weekly statistics on the web. Application support teams have access to top-notch performance data so they can see at a glance how their applications are faring. This is particularly vital when new code is being added. The support teams can see, for example, the before and after to determine if code is performing as expected. 

“We made dramatic code changes to an application, and the support team was anxious to see how successful this would be in reality,” said the IT manager. “It was easy for them to log onto Vityl Capacity Management to see that the changes had reduced CPU and memory utilization by 50 percent.” Vityl Capacity Management is also utilized to provide interactive, real-time reporting of servers, as well as to troubleshoot and create ad hoc reports. 

“One of the largest global players, and a key customer, recommended Vityl Capacity Management to help analyze ongoing performance issues during a customer operations review which was later described as a humbling experience due to the severity of the outage in an application.” 

“IT Service Analyzer lets us view how any server is doing at any time,” said the IT manager. “We can set alarms based on a variety of parameters and then dig in rapidly to see what’s really going on.” 

Vityl Capacity Management was utilized within the organization to analyze just how much time ongoing CPU enhancements would provide before further upgrades would have to be unleashed. Model made it a simple matter to take the company’s transaction traffic growth rate of 10 percent per month and model that into the future. As a result, IT estimates it has at least two more years left on those systems. 

“During the Olympics in China last summer, we had to be very sensitive to user trends,” said the IT manager. “We expected a lot of roaming and it was reassuring to know that our systems were ready for any spikes in demand.” 

Vityl Capacity Management was also used to manage projected growth. The previous year’s growth rate was 20 percent per month. For the current year, a monthly rate of 15 to 20 percent is predicted. In addition, a Tier 1 customer is likely to sign up for this service, which will exert a substantial spike in demand. Thus a new modeling scenario had to be run to compensate for this new situation. Results showed that IO could become constrained during the newly projected peaks. However, Vityl revealed that this only applied to two servers while the other eight involved with this application could cope with close to 1000% growth in traffic. 

Modeling of the application provided detailed information about how long the organization could last without upgrades, should management decide in favor of this path. Based upon no additional customers, it could last a year based on growth expectations using existing hardware. If two customers were added, on the other hand, two more server clusters would have to be deployed. If one new customer was added, solid state drives (SSD) would be needed to maintain transaction response times. 

“We were able to confine our hardware upgrades to a couple of servers,” said the IT manager. “In the past, we would probably have upgraded everything at a far greater expense.” He points out further financial benefits from modeling. By deferring the purchase of SSDs for at least a year, the company will be able to take advantage of much lower rates for those products as the prices are plummeting steadily. 

Vityl Capacity Management Support 

After two years of service from Vityl Capacity Management, how has he found the level of service and support available from the company? 

“Support has been awesome as people are always available to help me gain full value from the software,” said the IT manager. “They even made a major visit to our site AFTER we had become a customer. It is clearly a committed vendor.” 

The results from Vityl Capacity Management have spread around the company. Its German and UK data centers have adopted it. Further, the company has now adjusted its acquisition process. 

“Corporate finance won’t sign off on a hardware acquisition without a Vityl Capacity Management analysis model,” said the IT manager.