Prime Therapeutics Trusts Robot/SAVE With 7.5 Million Lives

Drugs. We all need them; we all use them. In fact, it’s said that prescription drugs are the most frequently used benefit of most health plans. So, who manages all those pharmacy services taking place out there?

One of the prime players in the pharmacy benefits industry is Prime Therapeutics, LLC. Based in Eagan, Minnesota, this company of approximately 1000 employees also has offices in Omaha, Chicago, and Irving, Texas. Prime Therapeutics provides pharmacy benefits management services to its clients, which include several Blue Cross/Blue Shield plans, employers, union groups, and other third-party administrators nationwide. The company numbers its member base at 7.5 million covered lives, and their claim volume at an estimated 60 million claims per year.

Terrie Morgan is the Manager of Production Control at Prime Therapeutics. She and Leigh Richardson, Senior Computer Operator, are responsible for computer operations on their IBM® iSeries™ systems. Their primary production server is an iSeries Model 825 in Texas. A second Model 825 is a three-way LPAR system located in Eagan. In addition, they have another iSeries that’s used for system administration and several Windows and UNIX servers. They run SXC RXClaims software for claims processing.

To manage their workload—from providing clients with a comprehensive pharmacy program to claims processing—the company relies on several operations automation products from Help/Systems, including Robot/ALERT,® Robot/CONSOLE,® Robot/CORRAL,® Robot/NETWORK,® Robot/SAVE,® and Robot/SCHEDULE.®

One of the most important IT functions in any business is having a reliable backup and recovery system in place. That’s where Robot/SAVE, the automated backup, recovery, and tape management package from Help/Systems, plays a vital role at Prime Therapeutics. Since the company was formed in 1998 through the merger of Pharmacy Gold, Inc. and ProPar Services, Inc., it has used Robot/SAVE for its backups.

All daily, weekly, and monthly backups are run using Robot/SAVE and two IBM 3494 automated media libraries (AMLs). One 3494 is located in Texas and the other, with two drives, can be found in Eagan. The company mirrors most of its data from the production system in Texas to the secondary system in Eagan. Thus, the majority of its data is backed up in Eagan. Only non-mirrored data is backed up in Texas.

Daily and weekly backups consist of a full SAVLIB of all libraries. According to Terrie, “We do a SAVLIB, instead of a save object, because, with the 3494, the 3592E tapes, and compression and compaction, we can hold about 700 gigabytes of data. So, it was to our advantage just to do a SAVLIB across the board.” The libraries included in the SAVLIB are defined to the backup.

Both daily and weekly backups have retention periods of 60 days. The main difference between the two is that the tapes used for the daily backups remain on-site in the 3494, while those used in the weekly backups are ejected from the AML and stored off-site. The weekly backups use a move set that automatically ejects the tape from the AML for off-site storage.

The production system in Texas is backed up remotely. Said Terrie, “We only have two network staff down in Texas. The only thing they have to do is load and unload tapes on Fridays and send them to [our storage facility]. They don’t have to be familiar with the iSeries; all they have to know is how to move the tapes in and out of the AML.”

Added Leigh, “[The 3494] has been a big life saver for us.”

The tapes in the 3494 are set up in a scratch pool so they are available for use after the data saved on them expires. Leigh explained, “That’s a really nice feature because you’re not unloading tapes. The tapes that we don’t send off-site just physically stay in the AML until they expire and then Robot/SAVE knows to use them again.”

The daily and weekly backups are scheduled in Robot/SCHEDULE, Help/Systems’ job scheduler and automatic computer operator. The first backup on the production system in Texas starts at 7 p.m. On the secondary system in Eagan, the company uses cross-system reactivity to back up the partitions to the AML.

Continued Terrie, “We share the tape drives on our secondary box in Eagan with our development partitions. We start our secondary box backup at 7 p.m. That’s split into two backups so that it runs on both drives. When one of them finishes, the drive is deallocated on the current partition and allocated on the development partition.”

Leigh added, “The first time we set it up, we had a few issues because it was too quick. We had to put a delay of 60 seconds in the backup set to allow it time to deallocate and reallocate the drive.”

Monthly backups are similar to the weekly backups; the tapes are ejected and sent off-site for a specified retention period. Currently, the company performs only its SAVLIB backups using Robot/SAVE. Its restricted state SAVSYS and NONSYS backups are still done manually just prior to an operating system upgrade.

Said Terrie, “We have 99.9% availability service level agreements we need to meet, so we keep the system up and running all the time. We only roll to our backup box if we’re doing something like an upgrade for our claims processing software. Our primary business is processing claims, so that application is replicated to our backup box. It only takes us about 10 minutes to role swap to the secondary box.”

Robot/SAVE’s restoration reports are sent between the systems in Eagan and Texas following each backup session. Said Terrie, “If we lost our Texas box, we’d have the information in Eagan as far as the tapes we’d need to restore the systems, and vice versa.”

Robot/SAVE’s object archive also has come in handy when they’ve needed to restore data. Explained Terrie, “Recently a developer needed to recover a library that contained some programs from the development box. He gave me the name of the library and the last time it was on the system. I just went into the object archive and looked it up—you can look it up either by library or object—and it told me exactly which volume it was saved on and when it was saved. With the 3494, I can just select option 3 in Robot/SAVE and it restores it for me. And, since we don’t have anybody down in Texas, it saves us a lot of time because we don’t have to wait for someone to load and unload a tape.”

Leigh continued, “Another thing that I like in Robot/SAVE is the add undefined libraries feature. Often new libraries are created—whether in production or development—that we’re not aware of. This feature ensures that we’re not missing any of them, because it automatically adds any undefined libraries to the backup set. You see that your backups are taking longer, and you can determine why from there.

“We also use Robot/SAVE’s alternate backup set feature, so we can append to the tapes. We’ve gone from…I think it was twenty-three 3590 tapes to one 3592E tape [per day]. That’s been really helpful as far as the storage and cost of tapes.”

When they first began using Robot/SAVE, the company used 3590 tape drives. When they decided to convert to the 3494 AMLs, they did a new installation of Robot/SAVE on the Texas iSeries. For the backup box in Eagan, they didn’t want to start out with a clean system because they had a lot of archived tapes and history. Instead, they performed a conversion and found that Help/Systems’ support staff was very helpful in walking them through the challenges they faced.

The Eagan LPAR system is configured in a data center using Robot/SAVE. The partitions share all the tapes in the AML. Prior to installing the AML, each system using the 3590 drives had tapes assigned to the specific system. According to Terrie, “That was kind of time-consuming.”

Continued Leigh, “So we went with the Data Center Management feature in Robot/SAVE…where the systems share all the tapes. That was kind of difficult to implement because everything had to be uniform across the board on all three systems before it would actually let you include these two other systems in the data center. We put together a project plan when we were going through the conversion and someone from Help/Systems’ staff actually walked through our project plan to see if we were missing anything.”

“It’s easier to manage tapes when you don’t have a different naming convention for each tape,” concluded Terrie.

The combination of Robot/SAVE and the IBM 3494 AML gives Prime Therapeutics the confidence that their important data will be available whenever it’s needed. And that, in turn, should make 7.5 million people feel confident that when they need medication, their claims will be processed quickly and reliably.