NSA

        A Sun special report


  System error


    The NSA has spent six years and hundreds of millions of dollars
    trying to kick-start a program, intended to help protect the United
    States against terrorism, that many experts say was doomed from the
    start.




January 29, 2006  Baltimore Sun

A program that was supposed to help the National Security Agency pluck
out electronic data crucial to the nation's safety is not up and running
more than six years and $1.2 billion after it was launched, according to
current and former government officials.

The classified project, code-named Trailblazer, was promoted as the
NSA's state-of-the-art tool for sifting through an ocean of modern-day
digital communications and uncovering key nuggets to protect the nation
against an ever-changing collection of enemies.

Its main goal when it was launched in 1999 was to enable NSA analysts to
connect the 2 million bits of data the agency ingests every hour -- a
task that has grown increasingly complex with the advent of the
Internet, cell phones, and instant messaging -- and enable analysts to
quickly pick out the most important information.

The stakes could scarcely be higher.

A major failure leading up to Sept. 11, 2001, involved communications
intelligence, investigators found.More than 30 hints of the impending
attack had been collected in the previous three years but had sat,
unnoted, in the NSA's databases, according to a joint congressional
inquiry into pre-Sept. 11 intelligence operations.

The NSA initiative, which was designed to spot and analyze such hints,
has resulted in little more than detailed schematic drawings filling
almost an entire wall, according to intelligence experts familiar with
the program. After an estimated $1.2 billion in development costs, only
a few isolated analytical and technical tools have been produced, said
an intelligence expert with extensive knowledge of the program.

Trailblazer is "the biggest boondoggle going on now in the intelligence
community," said Matthew Aid, who has advised three recent federal
commissions and panels that investigated the Sept. 11 intelligence failures.

Complex from the start - the initial Trailblazer plan called for more
than 1,000 priority items - the project ballooned as it was passed
through three separate NSA divisions, each with its own priorities,
former intelligence officials said. And, they said, Trailblazer's
overseers lacked either the influence or the time to clearly define
their goals and keep the project on track.

When the agency's inspector general looked at the NSA's handling of the
project in its first three years, it found in a 2003 report "inadequate
management and oversight" of private contractors and overpayment for the
work that was done, according to a recently declassified version of the
report obtained by The Sun through a Freedom of Information Act request.

Meanwhile, Science Applications International Corp. (SAIC), the lead
contractor on the project, did not provide enough people with the
technical or management skills to produce such a sophisticated system,
according to industry and NSA experts familiar with Trailblazer. And,
they said, the company did not say no when the NSA made unrealistic demands.

The company was initially awarded $280 million in 2002 to begin
construction.

SAIC spokesman Jared Adams declined to comment, saying, "We have been
asked to defer all comment regarding the NSA Trailblazer contract to the
NSA."

The reporting in this article includes interviews conducted over the
past three months with 25 intelligence professionals, 13 of whom worked
on or had oversight of Trailblazer. Because the program is classified,
most would not allow their names to be used.

Although the Bush administration spent much of the past week defending
the NSA's eavesdropping work as vital to keeping Americans safe from
terrorism, virtually no attention has been paid to the agency's failure
to deliver the system the NSA said was key to fulfilling that mission.

That means the government has been standing by while the agency has been
gradually "going deaf" as unimportant communications drown out key
pieces of information, a government official with extensive knowledge of
Trailblazer told The Sun.

NSA spokesman Don Weber said the agency would have no response to
requests for comment.

Listening in
Based at Fort Meade in Anne Arundel County and with field offices around
the world, the NSA harvests virtually every form of electronic
communication - including phone calls, e-mails, video links and bank
transactions - through a vast array of satellites, clandestine posts at
U.S. embassies, ground-based listening stations, and military airplanes,
ships and submarines.

The information collected and culled by the agency's approximately
40,000 employees accounts for an estimated 75 percent of the president's
daily intelligence briefing, said Aid, an intelligence consultant who is
writing a multivolume history of the NSA.

But there are huge holes in the agency's information filter. As a
result, a congressional report on 9/11 intelligence failures found,
"potentially vital" information is lost, particularly with regard to
terrorist groups.

That is what Trailblazer was designed to fix.

All digital communications trapped by the NSA are transmitted to the
agency's offices in computer codes of zeroes and ones. The sheer volume
of data being gathered is overwhelming the NSA's ability to digest it.
And that volume is growing every day with the advent of text messages,
hand-held computers such as the BlackBerry and phone conversations over
Internet lines.

The result is akin to looking for a needle in a haystack that doubles in
size every few months, said Aid, who has written extensively on
intelligence issues.

The agency has only blunt tools - largely based on the information's
origin or keywords linked to items of interest - to use in making
decisions about whether to keep captured data or discard it.

Intelligence experts familiar with the system said it is like deciding
whether to keep a piece of mail or throw it out based only on what is on
the outside of the envelope.

An estimated 95 percent of the information gathered is discarded without
being translated into an understandable form, said an intelligence
expert who has tracked the system for years. The remaining 5 percent,
still in the form of zeroes and ones, is turned into plain text or voice
recordings and routed to the appropriate division for analysis.

In each division, it might be run through software programs to identify
patterns or links with other data. But that is not guaranteed. Nor is
there a guarantee that a communication sent to a division dealing with
Latin America, for example, will ever be seen by an analyst tracking a
terrorist group that finances its activities through Latin American drug
smuggling.

NSA officials knew they needed to make changes in how they handled the
deluge of digital data and spent a year developing a broad concept for
how to do so.

As initially envisioned, said four intelligence experts with extensive
knowledge of the project, Trailblazer would have translated all of the
digital computer language (the zeroes and ones) into plain text or
voice. The data would have been analyzed to identify new patterns of
activity or connections among people whose communications are
intercepted, and then stored in an easily searchable database. Key
communications would have been automatically forwarded to the
appropriate analysts, who for the first time could have followed up with
their own searches of the database.

To implement Trailblazer, the NSA would have vastly expanded its
computing hardware and software, and made revolutionary changes in the
way huge amounts of data are stored and retrieved.

But years after the initiative was launched, there was still no
unanimity within the agency on how to achieve those goals, or even on
whether all of them were necessary or possible, interviews and records show.

A December 2002 report by the House and Senate intelligence committees
investigating pre-Sept. 11 intelligence failures found that although
Trailblazer was "frequently cited" as the solution to many of the NSA's
information management problems, "implementation of those solutions is
three to five years away and confusion still exists as to what will
actually be provided by the program."

The unclassified report also noted that without Trailblazer the NSA
employees with whom committee investigators spoke knew of no "near-term
efforts to alleviate their current system's technical limitations."

Another division of the NSA had been working on a separate, less
expensive program, code-named Thinthread. In development before
Trailblazer was launched, Thinthread tried to accomplish a similar goal
of separating the important communications from the junk.

A classified report from the Pentagon in 2004 found that Thinthread was
more promising than Trailblazer and could be put to use faster, said an
intelligence expert who was briefed on its contents.

NSA managers disagreed with the Pentagon report's conclusions and
canceled Thinthread, said the expert briefed on the report's contents.

As a result, nearly 4 1/2 years after the Sept. 11 attacks, the NSA
lacks a system to comprehensively evaluate all of the communications
collected by its vast networks of high-tech ears.

Blazing trails
Trailblazer began as a signature program of Gen. Michael V. Hayden, who
was the NSA's director from March 1999 until last spring. Early on,
former officials familiar with the program said, it became clear to
Hayden that the agency, with its rich history of developing cutting-edge
technology, was falling behind the technology curve. He cast Trailblazer
as the agency's future.

A cerebral Air Force general with close-cropped hair and wire-rimmed
glasses, Hayden saw his tenure as a key opportunity to turn the agency
around. In November 1999, he made Trailblazer a centerpiece of his "100
Days of Change" agenda.

Presented nearly two years before the 9/11 attacks, former colleagues
noted, Hayden's plan was prescient.

"It was going to structure us to handle the digital revolution," said a
former intelligence official. And, the official recalled, it would start
by building on the agency's existing computer systems.

But two months after Trailblazer was launched, the agency's computers
had a 3 1/2 -day meltdown. Hayden later described the episode in a 60
Minutes II interview as the agency's headquarters going "brain dead."

"It doesn't take a rocket scientist to know that an [information
technology] infrastructure that crashes ... is not going to be able to
handle Trailblazer applications," said a former intelligence official
familiar with the program.

Hayden declined, through a spokeswoman, to comment for this article and
referred questions to the NSA.

Several former intelligence and government oversight officials contend
that Trailblazer was doomed almost from its inception. The program
"kicked off with not a real great definition of what it was trying to
achieve," said a government oversight official, recalling an initial
briefing in December 1999.

Trailblazer began with such a burst of energy that it skipped some
crucial first steps, said current and former government officials close
to the program.

For example, to make sense of the communications it pulled in,
Trailblazer needed a standard format for all data so that it could sort
it properly, much like the standards Google uses so that it can search
different kinds of information on the Internet. But intelligence
officials said those standards were never defined.

The agency also boxed itself in by underestimating how long it should
keep old data, a former national security official said. As a result,
the system was designed to discard information that could later prove
useful, particularly in an open-ended war on terror.

Such early errors were exacerbated by the Sept. 11 attacks, which
prompted Hayden to push for faster implementation, eliminating time for
review and corrections, a former intelligence official said. And
Congress began throwing money at Trailblazer, discouraging a more
disciplined approach, said a former government official with extensive
knowledge of the program.

Monitoring project
While internal and external warnings that Trailblazer was going off
course were sounded, the extent of its problems gained little public
attention because the program was so secret and technical.

Since 1999, for example, more than 10 unclassified congressional reports
have pointed to "deficiencies" in NSA modernization efforts, but few
specifically pointed to problems with Trailblazer.

A 2003 NSA inspector general's report obtained by The Sun found that the
spy agency was unable to monitor the progress or the results of its
early Trailblazer contractors. Moreover, Inspector General Joel Brenner
said that his office could find no evidence of the program's specific
priorities, could not track the ways all of the money was being spent
and found that the NSA had overpaid some contract employees. The
contracts showed no limits on labor costs.

"These conditions are directly related to inadequate management and
oversight," Brenner's team said.

An intelligence expert who was briefed on a December 2004 report
conducted by the Pentagon's inspector general said the report found that
Trailblazer was not producing the system that had been promised and was
unlikely to produce it. The intelligence expert said the report
suggested agencywide management problems and recommended further
investigations of the NSA's overall acquisition and financial systems,
at least one of which has begun.

That report remains classified.

Shortly after the inspector general's review was completed, the NSA
hired IBM to take the lead on the project, said intelligence sources
familiar with the program.

Despite such warnings of problems, the Government Accountability Office,
the investigative arm of Congress that specializes in assessing program
management and government waste, has not looked into Trailblazer.
Randolph Hite, the GAO's director of information technology, said no one
in Congress has asked it to.

Former Sen. Bob Graham, a Florida Democrat who led the Senate
Intelligence Committee during and after the Sept. 11 attacks, said
Congress has failed to provide adequate oversight of the NSA.

"Most of the members don't have the background or the expertise to
understand very well an organization like the NSA," Graham said.

It did not help that Trailblazer, then less than two years old, was
faltering just as the country and Congress were attempting to cope with
the deaths of an estimated 3,000 people in the Sept. 11 attacks.
Congress was not interested in cutting any program related to fighting
terrorism for fear that it would be blamed if terrorists struck again.

 From that point, "our overwhelming focus was on trying to understand
that tragedy and the role the intelligence agencies had played," Graham
said. "Then, in 2002, summer and fall was the run-up to the war in Iraq,
so our attention was diverted."

That meant that on Capitol Hill, much of the oversight was left to the
few congressional staff members who understood the program. In July
2003, they persuaded their bosses to send the NSA a no-confidence
message about the agency's ability to manage complex programs such as
Trailblazer, said congressional aides.

Congress took away the NSA's authority to sign big-ticket contracts
without getting permission from the Department of Defense. The Defense
Department continues to be responsible for approving the NSA's proposals
to pursue and pay for large programs.

But that didn't stop or fix the project. At the NSA, Trailblazer
continued to stumble along.

Rep. C.A. Dutch Ruppersberger, a Democrat from Maryland who sits on the
House Intelligence Committee and whose district includes NSA
headquarters, said abandoning the concept behind Trailblazer is not an
option because "our national security depends on it."

"There was congressional oversight, and that's one of the reasons this
program has been red-flagged as a program that needs work," he said.
"The conclusion we all had was there were mistakes made, but the concept
has to move forward for the sake of our national security."

Sen. John D. Rockefeller IV of West Virginia, who succeeded Graham as
the senior Democrat on the Senate Intelligence Committee, said the
committee has "worried about the direction of the Trailblazer program
specifically and NSA acquisition practices generally" over the past few
years.

"We and the Armed Services Committee required detailed reports, withheld
money and ultimately removed the NSA's acquisition authority,"
Rockefeller said. "I expect that [NSA Director Lt. Gen. Keith B.]
Alexander will have a major restructuring plan to present to the
committee as part of the request for fiscal year 2007."

Turf battles
To understand where Trailblazer ended up, it is helpful to understand
the internal politics at the time of its launch.

The beginning of the Trailblazer program in 1999 coincided with a major
NSA reorganization. Deep inside the spy agency, knowledgeable
bureaucrats rushed to get their programs redefined as part of a project
favored by the director and presumably immune from budget cuts, two
former intelligence officials said.

As a result, they said, Trailblazer's scope mushroomed in the first few
years.

Meanwhile, the project was passed among several divisions, including the
new Transformation Office, which was shut down after a year; the Signals
Intelligence Directorate, which housed many of the analysts; and the
Information Technology Directorate, which builds technology systems.

Each time Trailblazer was moved, its new leaders altered its design.

"Every year or so, their story would be somewhat different about what it
is, what it's going to accomplish and how it is going to be
implemented," said a congressional aide who works with intelligence
programs.

The program landed in the lap of William B. Black Jr., the agency's
deputy director.

Black had spent four decades at the NSA before leaving it in 1997 to
join SAIC, a San Diego-based contractor with strong ties to the agency.
In 2000, Hayden called him back to become his top deputy and to take
charge of Trailblazer.

Two years later, the NSA awarded the prime contract to build Trailblazer
to SAIC, Black's former employer.

A careful bureaucrat who shies away from the media, Black was an expert
at navigating the agency's many fiefdoms and insisted that he make all
key decisions about Trailblazer, said intelligence officials with
extensive knowledge of the program. But they said Black had too much on
his plate to pay close attention to the program.

NSA spokesman Weber said Black was not available for an interview.

Lax internal oversight and shifting priorities quickly sent
Trailblazer's costs skyrocketing. In April 2005 Hayden testified before
Congress that the program, with publicly announced contracts then worth
$500 million, was "a couple to several hundred million" dollars over
budget and behind schedule.

Although he didn't provide details of the program's troubles, he
acknowledged in his testimony that getting the program off the ground
"was far more difficult than anyone anticipated."

New direction?
Five months into his tenure as NSA director, Alexander has been
reviewing Trailblazer, and he recently decided that he will try to
revamp it rather than scrap it, according to three intelligence experts
familiar with the program.

These officials and others knowledgeable about the program's history
said they were skeptical that Trailblazer could be fixed without
starting from scratch.

"Trailblazer is completely beyond fixing," said a former government
official who has tracked the program carefully. "Everybody who reviewed
Trailblazer after the first few months [of the program's launch] said it
was doomed or it should be scrapped."

Bobby Ray Inman, a former NSA director and a retired admiral, said there
needs to be some tolerance for altering the course of ambitious projects.

Several projects he considered successes, he said, were scaled-back
versions of an initial vision that "took us forward from where we were,
but it really didn't meet the aspirations of what we would have liked to
have had."

Alexander plans to hire a new executive to run the NSA's technology
programs, and Trailblazer will be one of this executive's top
priorities, said an intelligence consultant.

Alexander declined to comment for this article, but in August he told
The Sun that he would look to shift the agency's approach away from
large programs such as Trailblazer and toward smaller programs that
build on one another.

"I think the way to do it efficiently is smaller steps, more rapidly
done, rather than try to take one big jump and make it all the way
across," he said.

Those steps would involve significant changes in the way the NSA manages
data, including, he said, "how you handle data, how you visualize that
data and how we jump from Industrial Age analysis to the Information Age
analysis that our country needs."

Intelligence experts with extensive knowledge of the program said
Alexander is likely to salvage what he can from Trailblazer and largely
start over, casting it as a kind of "Trailblazer 2.0."

The country's new spymaster, Director of National Intelligence John D.
Negroponte, is taking on the job of connecting the technology systems of
all 15 intelligence agencies, and former intelligence officials said
Trailblazer's troubles should serve as a cautionary tale.

If Negroponte wants to learn the details, he won't have to go far. Since
last spring, his top deputy has been Hayden, the former NSA chief.

siobha..._at_baltsun.com
< http://www.baltimoresun.com/about/bal-reporterfeedback,0,4526743.htmlstory?recipient=siobhan.gorman_at_baltsun.com >