The military has a data problem. More specifically, it has a too-much-data problem. Analysts have to sort through massive amounts of information collected by orbiting surveillance drones and satellites, or finding the data trails left behind by spies inside defense networks. Sorting through all this data is also necessary for making unmanned vehicles more autonomous.
Bring on the White House’s new “big data” research initiative. Announced this morning, the plan aims to invest “more than $200 million” in six government agencies to develop systems to “extract knowledge and insights from large and complex collections of digital data,” according to a White House statement (.pdf). That means anything too large for normal software to handle, meaning data sets of at least dozens of terabytes, at minimum. The biggest beneficiary of all this could be the Department of Defense.
The Pentagon already spends hundreds of millions annually on “big data”-esque problems. The initiative announced today could add to that kitty up to $60 million per year for new research projects. That includes a $25 million yearly sum for a new Darpa data mining program called XDATA, which is broadly defined as a tool to analyze large amounts of meta-data and “unstructured” data like message traffic. (In comparison, the Department of Energy is receiving only $25 million in funding for a new data mining institute and the National Science Foundation is being granted $13.4 million.)