There's so much data even spies are struggling to find secrets
A staggering amount of data, from Facebook posts and YouTube clips to location pings from mobile phones and car apps, sits in the open internet, available to anyone who looks. US intelligence agencies have struggled for years to tap into such data, which they refer to as open-source intelligence, or OSINT.
Spying used to be all about secrets. Increasingly, it's about what's hiding in plain sight.
A staggering amount of data, from Facebook posts and YouTube clips to location pings from mobile phones and car apps, sits in the open internet, available to anyone who looks. US intelligence agencies have struggled for years to tap into such data, which they refer to as open-source intelligence, or OSINT. But that's starting to change.
In October the Office of the Director of National Intelligence, which oversees all the nation's intelligence agencies, brought in longtime analyst and cyber expert Jason Barrett to help with the US intelligence community's approach to OSINT. His immediate task will be to help develop the intelligence community's national OSINT strategy, which will focus on coordination, data acquisition and the development of tools to improve its approach to this type of intelligence work. ODNI expects to implement the plan in the coming months, according to a spokesperson.
Keep updated, follow The Business Standard's Google news channel
Barrett's appointment, which hasn't previously been reported publicly, comes after more than a year of work on the strategy led by the Central Intelligence Agency, which has for years headed up the government's efforts on OSINT.
The challenge with other forms of intelligence-gathering, such as electronic surveillance or human intelligence, can be secretly collecting enough information in the first place. With OSINT, the issue is sifting useful insights out of the unthinkable amount of information available digitally. "Our greatest weakness in OSINT has been the vast scale of how much we collect," says Randy Nixon, director of the CIA's Open Source Enterprise division.
Nixon's office has developed a tool similar to ChatGPT that uses artificial intelligence to sift the ever-growing flood of data. Now available to thousands of users within the federal government, the tool points analysts to the most important information and auto-summarizes content.
Government task forces have warned since the 1990s that the US was at risk of falling behind on OSINT. But the federal intelligence community has generally prioritized information it gathers itself, stymying progress. "You build your career on the idea that you have to obtain information covertly," says Senator Mark Warner, the Virginia Democrat who chairs the chamber's Intelligence Committee. "It's a mindset change to say, 'OK, no, I think we can learn just as much from open-source information.'"
Failing to develop new capabilities for using open data could be costly and even dangerous, say US policymakers and intelligence experts. OSINT is especially important when it comes to gathering information about the Chinese government, whose political system is highly compartmentalized and difficult to penetrate with human agents. Michael Morell, who served two stints as acting director of the CIA during the Obama administration, says identifying and making more open-source information available to analysts would significantly improve the performance of the US intelligence community.
The government is already working on OSINT. It used publicly available company records, procurement documents and satellite imagery to identify targets to sanction over alleged human rights abuses in Xinjiang, a territory in northwest China, according to a person familiar with the matter, who asked not to be identified discussing sensitive matters. The Department of State's Bureau of Intelligence and Research also used open-source methods to help identify the manufacturers of the Chinese spy balloon that traversed the continental US in early 2023, the person says.
By its nature, OSINT is also useful for groups beyond traditional intelligence agencies. In the months before Russia invaded Ukraine reporters and think tank analysts were able to verify claims of a Russian troop buildup using commercial satellite imagery, helping the Biden administration convince the American public that its warnings over Russia's plans to invade Ukraine were credible. Hamas has relied on television footage and social media posts to glean insights into the Israel Defense Forces' weaponry, drills and training, according to a May 2023 study in the journal Intelligence and National Security.
Houthi rebels have used Google searches and commercial shipping data to pinpoint vessels to attack, according to a person familiar with the situation. US officials say they believe China is supplementing its ability to track American naval operations by monitoring thousands of individual sailors' social media accounts, according to another person. Both sources asked to remain anonymous to discuss sensitive issues.
Despite some progress, multiple senior US policymakers describe the community's open-source efforts as insufficient. "I'd give the intelligence community a 'D' when it comes to its performance at open source," says Ellen McCarthy, who was head of the State Department's Bureau of Intelligence and Research from 2019 to 2021. "At the State Department, I saw policymakers going more and more to the private sector to get what they needed."
Part of the tradecraft of this kind of intelligence-gathering is doing it in such a way that targets of surveillance don't realize what's going on. If another government or organization realizes how the information it's making public is contributing to US intelligence, they might find a way to stop oversharing.
The potentially invasive nature of the work unnerves civil liberties advocates. US Senator Ron Wyden, a Democrat from Oregon, in December temporarily blocked the nomination of the next director of the National Security Agency until the agency disclosed to him whether it buys Americans' location and web-browsing data from commercial data brokers. In a letter to Wyden, the NSA said it acquires various types of commercially available information, which may include "information associated with electronic devices" but doesn't include location data for phones known to be used inside the country.
The government itself can be wary about some of the tools used to collect OSINT. At the CIA, for example, Russia analysts aren't allowed to use their desktop computers to access the social media app Telegram, which is popular among Russian military bloggers. The analysts are also barred from bringing personal devices into the workplace, forcing them to leave CIA premises when they want to access the app, according to people familiar with the situation, who requested anonymity to discuss CIA operations.
Coordination has also been a challenge. Different agencies gather data on their own, without necessarily sharing who's collecting what or tagging data in a way that allows them to combine their work and to have people with various levels of security clearance gain access to it. Rules regarding the collection and sharing of intelligence vary by agency, as does the definition of what even counts as OSINT. ODNI is set to issue revised definitions, as well as guidance on how it will treat commercially available information such as cellphone or web-browsing data.
(Bloomberg reported on Jan. 23 that the Biden administration is preparing an executive order to limit or prevent foreign adversaries from accessing personal data about Americans through legal means, such as purchasing it through data brokers.)
Power struggles within the intelligence community have also slowed progress on OSINT, critics say. "Everybody's so focused on controlling it, there's actually no one leading it," says Eliot Jardines, who oversaw the Open Source Center within ODNI before it was absorbed into the CIA in 2015.
As it stands, various parts of the intelligence community are pursuing their own operations. The National Geospatial-Intelligence Agency, which analyzes imagery from spy satellites, started an open-source project called Tearline in 2017 that works with universities and nonprofits on projects such as tracking the expansion of China's "Belt and Road" initiative.
Tearline's creator, Chris Rasmussen, argues for a standalone OSINT agency that's separate from the existing US intelligence community. He also wants OSINT reports to be delivered directly to policymakers' phones every morning—akin to the president's daily brief, but without classified information. The potential of this kind of information won't be truly realized until the process of gathering, analyzing and sharing OSINT is seen as something that stands on its own merits, he says, and "not just a supplement."
