Why mine data when you can smash it?

While most researchers are focused on carefully mining data, a pair of US researchers is set on 'annihilating' it.

Ishanu Chattopadhyay, research scientist at the Computation Institute and the Institute for Genomics and Systems Biology at the University of Chicago, US, and Hod Lipson, associate professor of mechanical engineering and of computing and information science at Cornell University, US, have devised a more efficient automated method of data stream comparison: data smashing. It highlights subtle distinctions — like irregular cardiac activity in digital heart recordings, or differences between speakers in a voice recognition database — and its results are on par with specialized algorithms and data mining processes devised by experts.

Both data mining and data smashing analyze relationships and patterns in data, but Chattopadhyay and Lipson's approach, published in the Journal of the Royal Society Interface, bypasses a significant hurdle: People. "Any time a data mining algorithm searches beyond simple correlations, a human expert must help define a notion of similarity by specifying important distinguishing 'features' of the data to compare, or by training algorithms using copious examples," notes Chattopadhyay.

In fact, a recent New York Timesarticle estimates that data scientists spend 50% to 80% of their time on data wrangling and cleanup. While this labor is mundane — and, as Chattopadhyay and Lipson argue, "slow, expensive, error prone, and unlikely to keep pace with the growing amounts and complexity of data" — sending raw data through an analysis algorithm rarely yields meaningful insights.

Data smashing, on the other hand, is based on the principle of 'information annihilation': Raw data streams (first quantized by assigning digits or symbols to continuous values) have corresponding anti-streams. When algorithmically collided, they reveal differences that are difficult to detect using conventional techniques. And, the process works without needing to understand the streams' sources or encoding.

By colliding data streams, then comparing them to noise, researchers can pursue complex new lines of thinking — without either domain knowledge or specialized training (they don't even have to know what they're after). Though better algorithms may exist for particular domains, data smashing accurately measures cause and effect in sequential observations, even in cases of missing or asynchronous data.

Chattopadhyay and Lipson applied data smashing to several distinct cases, including brain electrical activity. By smashing electroencephalographic (EEG) data streams from normal volunteers with eyes open and closed, and from epilepsy patients in seizure-free and seizure states, they detected clear clusters related to each state. They then defined a continuous curve including the four brain states. Ultimately, with no prior knowledge, they correctly identified epileptic pathology even in seizure-free states — developing a fuller, more insightful picture than possible with complex non-linear modeling.

Data smashing eliminates the pre-processing bottleneck and highlights details that are easy to miss using complex analysis techniques. For data novices, it removes the education and training barriers to realizing the significant opportunities big data offers. For data scientists, it gives back time to pursue game-changing discoveries.

To try data smashing, visit Chattopadhyay’s webpage.

- Sarah Engel