NIC network survives scare

On the afternoon of Feb. 2, an unwelcome and destructive digital guest made its way onto the North Idaho College campus.

That day, a computer virus began popping up around campus, and soon spread in an outbreak that lingered for nearly three weeks, bringing teams of computer technology specialists to the vanguard of the electronic realm.

“The question people always ask is, ‘Well how did it get here?’ And we don’t know that,” said Steve Ruppel, the college’s information technology director. “We wish we did, but in a sense it doesn’t matter anyway, because once it gets in, it spreads very quickly.”

How quickly? Within days after the first signs of infection, an estimated 90 percent of the college’s 1,200 faculty and staff computers had been infected – only the Apple Macintosh computers came through unscathed. The culprits were two sinister strains of the Virut virus, called Virut.bm and Virut.bh, which traveled through administrative login scripts and propagated as NIC faculty and staff tried to access their machines.

“It was bouncing through the network, so we were getting infected without knowing,” Ruppel said.

Normally when a virus infects one of the computers on campus, as they regularly do on a small-scale basis, the college’s two anti-virus and malware software systems would catch the perpetrator before it could spread. However, these particular Virut strains went under the software’s radar, explained Steve Smith, the college’s user services coordinator, and blocked them from downloading new security patches.

“That was kind of our litmus test to tell that” it was something different, Smith said. “When this thing ran over our existing defenses, it was like a free-for-all … we were at the tip of the spear with a new strain, something completely new. It took a day or two to figure out we were not fixing it – we didn’t have the tools in hand to fix it.”

Two days later, computer technology specialists spent most of their time peeling back the infected PC’s software layers, trying to catch a glimpse of the problem and how it spread. It was difficult work, since the virus hid among other detected problems, “so you don’t know which one’s the disease and which one is not,” Smith explained.

Meanwhile, the Virut strains had infected campus servers, further deteriorating the system and prompting the NIC administration to consider shutting down the network entirely to contain it.

It’s important to note, Ruppel added, that student and employee data was never compromised and classes went on as scheduled. Virut doesn’t corrupt data files, it just propagates through movable files and lowers anti-virus shields while simultaneously inviting in other viruses through malicious malware-loaded Web sites. While it could – and did in some cases – spread to portable thumb drives via infected machines, few if any student computers were affected.

As campus information technology teams strived to gain a foothold in containing the issue, they learned that NIC was one of only a dozen or so locations in the world to be suffering the effects of these particular Virut strains. The original virus, known as the Virut root kit, generally thought to have originated in Russia in 2006, and has since taken on more than one million unique forms on a worldwide computer blitz.

Finally on the fourth day, NIC’s information technology teams, working out of a makeshift crisis center in the Seibert building, identified it as Virut.

“Then we had something to research, we had something to ask questions about,” Ruppel said.

By the end of the week, the entire campus had gone into crisis mode. Administrative meetings were held in the morning and afternoons, with regular updates posted on all campus buildings. The computer technology team had grown to more than 20 members, including anyone Ruppel and Smith could find with technical background or know-how.

“There was a lot of stress. Things were kind of crazy, especially after we realized the extent,” Ruppel said, recalling those hectic first weeks during a recent interview, as he and Smith prepared a post-mortem analysis of NIC’s reaction. “You just don’t imagine that no one else has this.”

Also about the same time, the city of Houston began showing symptoms of the same virus, which city computer technology employees originally suspected as the headline-grabbing Conficker worm. It infected the Municipal Courts’ computer system, eventually spreading to more than 400 of the city’s 1,600 machines and causing officials to shut down court operations.

Houston officials chose to hire outside computer specialist company Gray Hat Research to identify and eliminate the virus.

“This is an extensive amount of work,” said Paul Williams, the company’s chief technology officer. “This is a very difficult virus. These viruses are so complex that you rarely can deconstruct it completely, you just have to try to contain it.”

On Feb. 6, five days after NIC first showed signs of infection, administration members decided to call for their own backup, just as they made the difficult decision to shut the network down.

Once Virut was discovered, “that’s when we started having conversations with Microsoft,” said Smith. “This is the biggest software company in the world and they were talking to us. It was pretty eye-opening that it was that big of a deal.”

Infected computer code was securely packaged and sent to Microsoft programmers at no cost, where they’d use it to write an updated patch to wipe out the virus. Microsoft gave NIC an estimated 24-hour turnaround time, and when that came and went Saturday night, “that’s when we were really getting nervous,” Ruppel said. Also, as a precaution Ruppel and Smith began calling every university, college and school in the area to let them know about the virus.

The Virut strains appeared limited to NIC’s campus, and no outside users were affected, Ruppel said. Some institutions sent their information technology teams to the area anyway to meet with NIC faculty members to share information about the virus, such as the Lewis-Clark State College Coeur d’Alene branch.

As the NIC computer specialist teams worked long into the weekend, some pulling 15-hour shifts, college faculty and staff members kept the workers awake by bringing in food, holding potlucks and supplying caffeinated beverages. The NIC network team had to rebuild the infrastructure in the middle of the night so that once they had the cure, “we had a safe and sterile environment to work in,” Smith said.

Ruppel was recently asked if virus writers just sit around, eating pizza and drinking coffee while they create insidious code. “I don’t know if that’s how they do it, but that’s how we fight it,” he said with a laugh.

After a tense few hours, late that Saturday night, the Microsoft patch finally appeared.

But even with a fix in hand, the ground work was just beginning.

“We had to become a virus recovery team, we had to gear up for mass production” Ruppel explained, referring to the intricate recovery process he and Smith had outlined over the weekend in how to deal with the infection now that they had Microsoft’s patch. They were even given the go-ahead by NIC President Priscilla Bell to tag each door in every campus building with posters declaring the computers were not yet safe for use.

CDs with the fix had to be burned and disseminated to the various recovery teams, which included more than 40 people by the start of the second week. Team leaders were learning new tips and tricks on the fly in cleaning each machine, which could take more than two hours, and then filtered them down to other team members so they could clear the campus as quickly as possible. A complex cleaning instruction sheet went through several iterations by the end of the week, which featured a tiered system for buildings by order of importance, and green or red labels were placed on every campus computer indicating which were secure and which needed more work.

“We really had to interject ourselves into the management structure here and say you can’t work on your computers, and here’s why,” Ruppel said.

About the red and green labeling method, Smith added, “This wasn’t something we could do in a logical fashion. This was a brute force method – we had to get an army of people out.”

As if things couldn’t get more chaotic, Smith and his wife welcomed a new baby daughter, Payton Elizabeth, just after 2 a.m. on Feb. 17. “That really lifted everybody’s spirit,” Ruppel said, referring to the exhausted recovery group members.

After almost three weeks of sweeping the campus, the final infected machines were cleared in early March. Green posters proclaiming “I survived Virutastrophy,” as the event was dubbed by campus faculty and staff members, were placed on doors and walls around campus, many of which still hang today.

And even though the college saved money by deciding to fix the issue in-house, the episode cost NIC hundreds of hours of student and faculty productivity.

Some important lessons the college has learned, Ruppel said, is if there is another virus outbreak they won’t let users continue to work on their machines, which they now know helped spread the virus. Overall though, he said the entire campus pulled together during a trying time.

“It was a total team effort, we became one team and everyone worked together,” Ruppel said. “I never thought I’d go through anything like this. You don’t see it coming; you know it can happen; it hits and now you have to respond.”

And, Smith added, “I know we would be prepared to handle it if the worst happened. We have a great group of people here.”