The Internet is truly all-present today. It contained 4.4 zettabytes of data as of 2013, and by 2020 is expected to hit 44 zettabytes. However, a huge portion of that data –an estimated 70% to 80%– is unstructured, and therefore not useful for companies that process and analyze data. This unstructured data, known as “dark data” includes things such as images and text.
Recently, companies have begun exploring how to use AI, or artificial intelligence, to unlock some of the information contained in this dark data. AIs would be able to find useful information and convert it into more useful formats. To this end, computer scientists at Stanford created DeepDive, a tool that could extract dark data.
Then, in 2015, MacArthur Genius Grant recipient Christopher Ré, along with Michael Cafarella, Raphael Hoffmann, and Feng Niu, founded Lattice, a company dedicated to the concept of data extraction with AI. Lattice was recently acquired by Apple, which hopes to expand into the artificial intelligence market. Prior to its acquisition, Lattice went largely unnoticed by the general public, although it certainly didn’t escape the gaze of tech companies. Lattice is at the forefront of AI development, and has even been talking with companies like Amazon and Samsung about how to improve their own artificial intelligence programming.
Why is it so important to be able to analyze dark data? Today, data is a multibillion dollar industry, and everyone from start-ups to world governments are trying to get in on it. It can be used in everything from advertising and medicine to international policing and paleontological research. The net has become a place where major financial transactions take place, where global market trends become apparent, and where crimes are committed and solved. More data means that the companies that analyze this data can see patterns more easily, and see these patterns on a broader range of subjects. Structuring dark data would make analyses much more effective.
Meanwhile, more and more companies are investing in artificial intelligence. The AI market value is expected to double this year, from 6.43.7 million dollars in 2016. The increased accessibility of data likely to result is bound to raise concerns about internet privacy, but also a slew of exciting possibilities for the coming years.