Subscribe now
Special Series

Bugs in the System

How a bungled software rollout is endangering Inland Northwest veterans

Published: Jan. 11, 2023

It was supposed to be a major update of 40-year-old software that keeps tracks of patients' medical records and allows their doctors at the Department of Veterans Affairs' roughly 1,300 clinics and hospitals to order tests, prescriptions, and follow-up visits.

What's more, the new system would be compatible with a new system being installed by the U.S. military.

What it was, however, was a failure from the start. The system didn't work as it was supposed to. Care to veterans was delayed. Patients missed important medications. The system crashed repeatedly, creating more work for medical providers. Promises for fixes weren't kept. Officials misled Congress and the public about the severity of the problems.

Two years after it was installed at Spokane's Mann-Grandstaff VA Medical Center, the Cerner Millennium software still doesn't work well.

Here's a recap of the entire Cerner Millennium saga, as covered by The Spokesman-Review's Orion Donovan-Smith, Arielle Dreher, Thomas Clouse, Kip Hill, design editor Charles Apple, and others.

* Reporting by Orion Donovan-Smith and Arielle Dreher was funded by the Report for America program and can be republished by other organizations for free under a Creative Commons license.

Broken Promises

The Department of Veterans Affairs was once hailed for pioneering the field of electronic health records – computer systems that track patient information and help doctors and nurses coordinate health care while reducing the risk of medical errors. But by the time Donald Trump became president in early 2017, the narrative had changed. The VA’s aging VistA system, despite remaining popular with its users, was under fire because it couldn’t easily share data with the system the Department of Defense used in its medical facilities, causing delays and headaches for service members when they left the military.

The solution, Trump’s son-in-law Jared Kushner decided after briefly consulting with VA and Pentagon officials, was seemingly simple: The VA should use the same system the Department of Defense had bought in 2015, made by a company called Cerner Corporation, which was first deployed at Fairchild Air Force Base outside Spokane in February 2017. When then-VA Secretary David Shulkin announced in June 2017 that his department would skip the usual competitive bidding process and give Cerner a $10 billion contract to buy the same system being adopted by the Pentagon, Trump called it “one of the biggest wins for our veterans in decades.”

Read the story:

More than two years after the VA began testing the system at Mann-Grandstaff VA Medical Center in Spokane and its affiliated clinics across the Inland Northwest, that promise hasn’t been borne out. Keep scrolling to see how a project meant to improve health care for veterans has instead harmed them and pushed VA employees to quit, according to The Spokesman-Review’s reporting.

The Rollout

Calif.Nev.Mont.Wyo.UtahColo.Ariz.Neb.S.D.N.D.Minn.IowaWis.Ill.Ind.OhioMich.Penn.N.Y.N.M.TexasOkla.Kan.Mo.Ark.La.Miss.Tenn.Ala.Ga.Fla.Ky.Va.N.C.MaineAlaskaHawaiiS.C.Ore.IdahoBremertonLakewoodFairchild AirForce BaseOak HarborSpokaneWenatcheeCoeur d'AleneLibbySandpointSpokaneWalla WallaRoseburgWhite CityColumbusDaytonAnn ArborSaginawPuget SoundCincinnatiPortlandAnchorageIron MountainMilwaukeeTomahMadisonBattle CreekDetroitBoise

  1. Pentagon Deployment

    In 2017, the Defense Department rolls out the Cerner software for its medical centers at four military bases in Washington, starting with Fairchild Air Force Base. The following year, the department sums up its experience so far by stating that the software is "neither operationally effective nor operationally suitable" and that it "does not demonstrate enough workable functionality."

  2. VA Testing

    The Veterans Affairs administration announces in June 2017 that it, too, will use the Cerner software for its electronic records. In October 2018, they announce that the pilot project will be Spokane's Mann-Grandstaff hospital, along with several clinics in the region attached to Mann-Grandstaff. After multiple delays, the software is launched in October 2020.

  3. Troubled Expansion

    Despite the troubles experienced at Mann-Grandstaff, the protests of several members of Congress, and some lengthy delays, the VA continues rolling the Cerner software out to other medical centers across the country throughout 2020 and 2021.

  4. Delays

    Medical centers in these cities were scheduled to launch the Cerner software between January and March of 2023, but the launch is pushed to June in light of both the continued troubles at centers using the software and Cerner's acquisition by tech giant Oracle in June 2022.

Original print graphic by Molly Quinn | Adapted for the web by Joel Smith

Guinea Pigs

When the VA launched the new computer system at the Spokane hospital and its affiliated clinics across the Inland Northwest, a top Trump administration official said it “went extremely well.” By the time President Joe Biden took office three months later, employees at Mann-Grandstaff knew that wasn’t true.

New VA Secretary Denis McDonough ordered a review of the project in March 2021 and visited Spokane a month later to hear concerns from veterans and VA staff, and in July 2021 he called the review’s findings “extremely disappointing.” The Biden administration reversed many of the policies enacted under former President Donald Trump, but that didn’t happen to the VA’s rollout of the Cerner system.

By the end of that year, top VA officials were expressing optimism and pushing to bring the system to more facilities, including the medical center in Walla Walla. The VA’s deputy secretary told lawmakers in November 2021, “The Cerner system works and I believe we’ve properly positioned it for success.” An hour later, the system went down for the 11th time in two months.

Flaws with the system had made about two-thirds of employees consider quitting, according to an internal survey, and made veterans – many of whom were exposed to toxic substances and other risks during their military service – feel they were being used as test subjects.

Some of them chose to speak out in The Spokesman-Review’s first major story on the problems:

Assuming the Worst

By March 2022, VA employees in Spokane were used to the Cerner system frequently crashing or being only partly usable, but they knew something bigger had happened when the hospital’s director sent an email telling them to stop using the system and to “Assume all electronic patient data is corrupted/inaccurate.”

An overnight update to the system, a VA official explained later, had “jumbled” data and left some patient records with information from the wrong veterans. No new patients could be admitted to the hospital for a full day, although health care providers saw patients who had already been admitted, scribbling notes on paper in lieu of entering them into the computer system.

In May 2022, the VA confirmed the system had been partly or completely unusable at least 50 times since launching in Spokane, and records obtained by FedScoop in August 2022 showed the total number of such "major incidents" was nearly 500.

How the System Fails

The Cerner Millenium system was designed to keep track of patient information and to allow medical caregivers to send orders requesting X-rays, blood test, appointments with specialists, and prescriptions.

But here's what happened instead...

  1. After seeing a patient, a doctor wanting the veteran to return for a follow-up appointment would enter the patient's information in the Cerner system and be prompted to choose a location from a complicated drop-down menu with confusingly named options like "668QD SW" and "668QB SW."

  2. Rather than sending the order along, some of the requests would result in an error and would reroute the order to an unmonitored holding area, which came to be known as the "unknown queue."

  3. To make matters worse, the order would appear to have been sent successfully. The system would have given no indication of the error, leaving doctors with no reason to investigate, and allowing the order to fall through the cracks.

  4. Missing orders could go overlooked by health care providers for days, weeks, or even months, delaying veterans' care. Thousand of such orders piled up in the "unknown queue", which VA officials called feature, not a flaw.

Adapted from a print graphic by Charles Apple

Harm Comes to Light

Despite bipartisan calls from Washington’s representatives in Congress to delay the system’s further rollout until problems were fixed – something McDonough had promised to do – the VA launched it March 26, 2022, at a hospital in Walla Walla and its affiliated clinics in Washington, Oregon and Idaho.

A month later, The Spokesman-Review reported a veteran had been hospitalized in Spokane with heart failure after a problem with the Cerner system caused a vital medication to be mistakenly stopped. The incident, which happened more than a week before the system launched in Walla Walla, was serious enough that leaders at Mann-Grandstaff reported it as a “sentinel event,” a designation reserved for cases of potentially severe harm that prompted an investigation.

VA Delays Rollout After Spokesman-Review Reveals Widespread Harm

In an April 26 hearing, House lawmakers cited The Spokesman-Review’s reporting on the hospitalization in Spokane and told VA officials the system had to be fixed before being deployed at larger, more complex hospitals such as the VA medical centers in Seattle, Portland and Boise, where it was scheduled to launch that summer.

The VA executive in charge of the project told the panel the heart failure in Spokane was caused by “a usual thing that happens” in the system and suggested staff in Spokane should have known how to prevent the error. Two days later, the system launched in Columbus, Ohio, its first site outside the Northwest.

After the report of harm to a veteran in Spokane, McDonough told reporters April 25 he was not aware of any other cases of harm related to the new system. But on June 19, a draft report by the VA Office of Inspector General obtained by The Spokesman-Review revealed that an investigation ordered by the secretary after he visited Spokane in April 2021 had found nearly 150 cases of harm caused by a single flaw in the system.

Read the story:

The watchdog agency’s draft report, which was released a month later, also found that VA Deputy Secretary Donald Remy had been briefed on the cases of harm in October 2021 and had shared the information with the official in charge of the project, Terry Adirim. Neither official had mentioned the harm in subsequent testimony before Congress, and McDonough has maintained he was unaware of the findings of the inquiry he had ordered.

After The Spokesman-Review notified the VA it had obtained the report, the department didn’t reply but instead postponed the system’s launch in Seattle, Boise and other cities.

In June, the tech giant Oracle acquired Cerner in a deal worth more than $28 billion, turning the health technology company into a subsidiary known as Oracle Cerner.

Human Collateral

Charlie Bourg, an Army veteran from Chewelah, had been an advocate for patients and employees at the Spokane VA for years, so it didn’t take long for him to hear about problems with the Oracle Cerner system. He worried its flaws could harm veterans, but he didn’t expect to be one of them.

The 68-year-old grandfather learned from doctors in May 2022 that he had terminal prostate cancer. Had it been caught sooner, a doctor told him, it may have been treatable, but problems with the new computer system had caused a yearlong delay in diagnosis and treatment when referral orders were routed incorrectly.

Read the story:

Despite reports of similar problems from every site where the Oracle Cerner system had been launched, including hospitals in Ohio and Oregon, VA officials said they planned to continue the system’s rollout in January 2023.

Quantifying the Problem

498
Number of "major incidents" with the Cerner system between Sept. 8, 2020 and June 10, 2022 (a)
1,072
Number of hours when the system was partly or completely unusable between in that same time period (b)
$32.7B

Cost of implementing the Cerner system over 13 years — more than double the VA's previous projection (c)

2

Cases of "major harm" the VA Office of Inspector General attributed to the “unknown queue,” just one of 60 different safety concerns identified by a VA patient safety team. (d)

52

Cases of “moderate harm” the OIG attributed to the “unknown queue.” (e)

95

Cases of “minor harm” the OIG attributed to the “unknown queue.” (f)

a.
Credit: FedScoop
b.
Credit: FedScoop
c.
According to a cost estimate completed in October 2022 by the Institute for Defense Analyses at the VA's request
d.
The Veterans Health Administration defines major harm as causing “permanent decrease in the body’s functioning or disfigurement, requires surgery or inpatient care.”
e.
Defined as causing “increased length of hospital stay or required increase in level of care.”
f.
Defined as causing no injury, no increased length of stay and no increased level of care.

Widespread Repercussions

After the Senate confirmed an official to lead the Veterans Health Administration for the first time since Barack Obama was president, that official, Shereef Elnahal, told The Spokesman-Review in an exclusive interview the VA would postpone the Oracle Cerner system’s launch by at least another six months and was notifying more than 41,000 veterans their care may have been delayed by problems with the system. That number later rose to more than 71,000 veterans.

Read the story:

At the same time, the VA confirmed to The Spokesman-Review that a patient of the clinic in Columbus had died in September 2022 in an incident that was being investigated as a potential “sentinel event” linked to a medication error.

Fallout at Mann-Grandstaff

The VA confirmed in September the Oracle Cerner system had caused a budget shortfall at the Spokane hospital, because it required hiring more employees to make up for lost productivity and reduced revenue due to billing problems and treating fewer patients. But a VA spokesman said the budget problems would not lead to reduced staffing levels at the hospitals affected by the new system.

Contrary to that claim, an investigation by The Spokesman-Review found that despite ongoing problems with the system, regional VA leaders had pushed the Spokane hospital to reduce its staff in an effort to balance the budget. At the same time, reduced productivity at Mann-Grandstaff caused by the system had forced more veterans to seek care in private clinics and hospitals, where an internal email revealed they could wait months to get the treatment they needed.

More than two years after the VA began testing the system in Spokane, more than 71% of Mann-Grandstaff employees said in an internal survey that it had worsened their level of burnout and nearly 60% were considering quitting because of the system. Nine current and former employees chose to speak out, saying their frustration with the system outweighed their concerns about retaliation.

Read the story:

At the end of 2022, the Oracle Cerner system was scheduled to launch at hospitals in Michigan in June 2023 but the rollout at other facilities in the Northwest was delayed until 2024. In the meantime, veterans and VA employees across the Inland Northwest, southern Oregon and central Ohio continue to deal with the consequences of a project that could cost nearly $50 billion, according to an estimate commissioned by the VA.

The Issue in Print

The Story Continues

All continuing coverage

Timeline of Events

1996

  1. An advisory committee set up by President Bill Clinton recommends the Department of Defense develop a "mechanism for computerizing medical data" and says the Pentagon and the Department of Veterans Affairs "should adopt standardized record-keeping to ensure continuity.

    The VA is using an electronic healthy system called VistaA, developing in house and launched in 1978. The DoD is using a separate system later renamed AHLTA.

2008

  1. Congress directs the DoD and the VA to develop health record systems with "full interoperability" and to set up an office to coordinate that effort.

2013

  1. After years of interdepartmental bickering, the effort to pursue a shared system is abandoned when the Department of Defense opts to buy a commercial health record system.

    In frustration, Congress again mandates the DoD and VA come up with a system with an "integrated display of data or a single electronic health record."

2015

  1. After an open bidding process, the Pentagon awards a $4.3 billion contract to a group of companies — including Cerner of Kansas City, Missouri — for an integrated health record system.

2017

  1. February

    The Department of Defense launches the Cerner Millennium health record system at Fairchild Air Force Base and then begins rolling it out to other bases in Western Washington. It isn't long before reports surface of missing prescriptions, lost referrals, and system crashes.

  2. March

    President Donald Trump puts his son-in-law, Jared Kushner, in charge of a new White House effort to infuse government with business ideas. One of his first projects: integrating medical records of the VA and the Department of Defense.

  3. June 5

    VA Secretary David Shulkin announces his department's electronic health record system will be replaced by one by Cerner. Trump praises the system, saying it will mean "faster, better and far better quality care" and calls it "one of the biggest wins for our veterans in decades."

    The contract to replace the VA system was awarded without going through a competitive bidding process. The justification: The Cerner system is the same one the DoD is switching to.

2018

  1. March

    Shulkin leaves his job, claiming Trump has ousted him for resisting efforts to privatize the VA. The next month, Trump nominates Robert Wilkie — who was overseeing the Pentagon's rollout of the Cerner system — as acting VA secretary.

  2. May 1

    Wilkie signs the $10 billion contract with Cerner, despite a Pentagon report in April finding the system was riddled with problems and was neither "effective" nor "suitable". The next day, Trump nominates Wilkie to serve as permanent VA secretary.

  3. Oct. 17

    During a visit to Fairchild Air Force Base, Wilkie announces the installation and testing process of the new Cerner system is to begin at Spokane's Mann-Grandstaff VA Medical Center in 2020. It will then be rolled out nationwide to all VA centers by 2028.

  4. Nov. 29

    The Department of Veterans Affairs says it will delay the opening of a 24-hour urgent-care center at Mann-Grandstaff until the system upgrade is up and running.

2020

  1. Jan. 29

    Sen. Patty Murray, who's on the Senate Veterans' Affairs Committee, urges the VA executive in charge of the system rollout to address staffing and facility concerns at Mann-Grandstaff.

  2. Feb. 3

    Wilkie announces he has dismissed VA Deputy Secretary James Byrne, who had been overseeing the system rollout, "due to loss of confidence in Mr. Byrne's ability to carry out his duties." Later that month, Wilkie takes on leading the Cerner project himself.

  3. Feb. 12

    The VA requests an 82% increase in spending in 2021 for its electronic records modernization project. The next day, the VA says it will push back the rollout of the Cerner system — which had been planned for the next month — in order to make more time for installation and training.

  4. April 27

    A Government Accountability Office report reveals that three members of Donald Trump's Mar-a-Lago resort had advised on various VA projects including its contract with Cerner.

  5. Oct. 23

    The VA finally launches the Cerner Millennium System at Mann-Grandstaff. It's what developers call "capability set 1.1" — essentially, an unfinished version that doesn't include some functions needed at larger hospitals.

    Staffers say later they weren't trained on some parts of the system because they weren't working yet. They said they had been assured those functions would be ready on time.

2021

  1. Feb. 11

    The Government Accountability Office recommends the VA postpone the use of the Cerner system at any other medical center until problems that are surfacing at Mann-Grandstaff have been addressed.

  2. March 19

    Newly installed VA Secretary Denis McDonough orders a "strategic review" of the Cerner system and its reported issues.

  3. April 14

    Mann-Grandstaff officials says employees have raised 247 patient safety concerns related to the new system. Primary care productivity had declined by 38% at one point after the launch, officials say. VA officials tells a House subcommittee it would delay using the system elsewhere in the U.S.

  4. June

    Congress approves another $2.6 billion for the Cerner rollout. New estimates are that the new system system will cost a total of $21 billion — original estimates had been $16 billion.

  5. July 14

    Following a 90-day review, McDonough tells a Senate Committee he's halting further hospitals from adopting the new system. He says the problems have been caused by bad management and planning and not by the software itself.

  6. Nov. 2

    "The Cerner system works," VA Deputy Secretary Donald Remy tells a House subcommittee. "And I believe we've properly positioned it for success."

    An hour later, the system at Mann-Grandstaff crashes and doesn't come back up for 80 minutes.

    VA employees of Mann-Grandstaff have made 576 reports of patient safety issues related to the new system. Two-thirds of the staff at Mann-Grandstaff say they are exhausted, burned out and are considering quitting.

    Despite McDonough's earlier promise not to roll out the system to other VA hospitals until issues at Mann-Grandstaff had been fixed, the VA confirms it plans to begin using the system in Columbus, Ohio, in February. After that, the system will be brought to Walla Walla and its affiliated clinics.

  7. December

    In a Spokesman-Review report, veterans and current VA employees describe a system that slows treatment and threatens patient safety, despite VA officials calling the transition "flawless."

  8. Dec. 20

    Austin, Texas-based software giant Oracle announces it will acquire Cerner for about $28.3 billion.

2022

  1. Jan. 14

    The VA announces it will delay further deployment of the Cerner system in Columbus until the end of April due to a surge of COVID-19 cases there.

  2. Feb. 3

    Rep. Cathy McMorris-Rodgers asks the VA to delay rollout of the Cerner system in Walla Walla until the ongoing problems have been addressed.

  3. March 3

    Work halts at Mann-Grandstaff when "a code defect" in an update to the Cerner system corrupts data for more than 200 patients. Admissions are halted and ordering medications, lab tests and imaging requests is limited. Employees are told to "assume all electronic data is corrupted/inaccurate" and to write down information to enter it into the system later.

    The system is restored 20 1/2 hours later, although some records remain inaccessible throughout the next day. VA officials later say all patient data was restored by the end of the following week.

  4. March 15

    After a veteran is admitted to Mann-Grandstaff for heart failure, staffers discover a prescription for heart medication that had been prescribed before the launch of the Cerner system had expired but wasn't show in the system. Reports released two days later cite 21 different medication-related problems reported by employees.

  5. March 22

    Remy says the VA will go on with launching in Walla Walla later that week. He says the number of patients seen each day at Walla Walla and its affiliated clinics will drop but would return to normal about three months.

  6. April 25

    McDonough says the Department of Veterans Affairs will not need to request more money from Congress to complete the rollout of the Cerner system. This comes after an Inspector General's report that the new system would cost an additional $2 billion for each year the rollout runs behind schedule.

    McDonough tells The Spokesman-Review he's not aware of any other cases — besides the heart failure patient at Mann-Grandstaff — in which a veteran has been harmed in connection to the Cerner system.

  7. April 26

    Remy visits Mann-Grandstaff. During his visit, the Cerner system crashes and remains down for about 45 minutes. It crashes again the next day during a House VA subcommittee hearing about the system.

  8. May 8

    The VA confirms the Cerner system has had 42 "unplanned degradations" and eight "unplanned outages" — for a total of 30 hours and 33 minutes — since its launch in 2020, not counting the two outages on April 26 and 27. Six such outages have taken place in just the past two months, the VA says.

  9. May 27

    Mann-Grandstaff announces that construction over the weekend would likely cause a 12-hour outage of the Cerner system. The hospital says the outage won't affect patient care.

  10. June 1

    A watchdog report reveals Mann-Grandstaff has failed to collect and report data needed to maintain the hospital's accreditation since launching the Cerner system 20 months before. Of 103 performance metrics monitored by the VA, only 13 are available from the Cerner system, the report says.

  11. June 9

    Oracle's acquisition of Cerner is finalized. Oracle founder Larry Ellison promises to modernize the Cerner Millennium system and to "make it much easier to use."

  12. June 17

    VA officials say the scheduled rollout of the Cerner system in Seattle and elsewhere in the Puget Sound region, scheduled for August, will be pushed back until March 2023. Four days later, the VA says it'll delay the rollout in Boise, planned for the following weekend, until July.

  13. June 18

    The Spokesman-Review reports that a VA draft report says the Cerner system has failed to deliver more than 11,000 orders for specialty care, lab work, and other services and has caused harm to at least 148 veterans in the region.

    This is after McDonough said he was not aware of harm to patients caused by the system and would halt its rollout to other centers if it was determined it increased risk to veterans. The report further says Remy had been informed of specific cases of harm in November 2021 and was told the problems were not fixed.

    Four days later, McDonough announces his confidence in the Cerner system has been shaken.

  14. July 19

    The VA Office of Inspector General releases its report of an internal investigation of the Cerner system rollout that The Spokesman-Review had revealed the previous month. A second report show VA leaders in charge of training had misled investigators about how many employees had passed readiness tests before the system launched.

    Documents obtained by The Spokesman-Review reveal the Cerner system has crashed much more often than had been previously reported by the VA. The system had been degraded, suffered "down-time," or had full or partial outages more than 180 times just since the previous September.

    An Oracle official says she's confident the VA system will become "the standard bearer for the industry" and that "nothing we have discovered" since buying Cerner "has caused us to waver form that conviction."

    A day later, the VA announces it will again delay rollout of the Cerner system in Boise.

    In a Senate hearing the next day, it's revealed an analysis shows the system rollout — which was supposed to cost $16.1 billion over 10 years — is now expected to cost $33.6 billion over 13 years and then $50.8 billion to sustain over 28 years.

  15. Sept. 21

    Remy admits to the Senate Appropriations Committee that the Cerner system "needs major improvements."

    An Oracle executive tells the committee that Oracle will rewrite the Cerner system as a new, cloud-based application that will address many of the problems with which providers at Mann-Grandstaff have struggled. He says a beta version of the new system will be delivered as a "cost-free upgrade" to the VA and would be available in 2023.

  16. Sept. 28

    VA Undersecretary for Health Shereef Elnahal says that budget shortfalls resulting from the Cerner rollout will not result in cuts to service at Mann-Grandstaff or other hospitals using the Cerner system.

  17. Oct. 13

    The VA announces it will again postpone rollout of the Oracle Cerner system at other hospitals.

  18. Nov. 22

    The National Oceanic and Atmospheric Administration announces it will join the joint Oracle Cerner system in the summer of 2023.

  19. Dec. 9

    A VA official says the VA will hire 1,000 new IT workers to help with the rollout of the Oracle Cerner system.