Imagine an advanced artificial intelligence (AI) system that can view and interpret moving images with the sophistication of the human mind. Scientists at Scripps Research have made this vision a reality by developing MovieNet, an advanced AI that analyzes video in a similar way to how our brains process real-world environments as they unfold.
While traditional AI systems excel at identifying static images, MovieNet pioneers a new approach to machine learning models for understanding complex, evolving scenes, a breakthrough with significant implications for areas such as medical diagnostics and autonomous driving, where detecting subtle changes over time is essential. Compared to traditional AI, MovieNet is not only more accurate but also more environmentally friendly. “The brain does more than just look at pictures; it creates a continuous visual narrative,” says senior author Hollis Kline, PhD, director of the Dorris Center for Neuroscience and the Hahn Professor of Neuroscience at Scripps Research. “Static image recognition has come a long way, but processing continuous scenes, such as watching a movie, involves a much more complex process of pattern recognition. By studying how neurons understand these sequences, we were able to apply similar strategies to AI.”
In developing MovieNet, Klein and lead author Masaki Hiramoto, a researcher at Scripps Research, studied how the brain perceives real-world scenes as short sequences that resemble movie clips. They specifically studied how tadpole neurons respond to visual stimuli. “Tadpoles have a remarkable visual system, and we know that they can detect and respond to motion effectively,” Hiramoto explains.
He and Klein identified neurons that respond to movie-like features, such as changes in brightness and image rotation, and can recognize objects in motion. These neurons, located in a visual processing area of the brain known as the optic tectum, assemble the pieces of a moving image into a coherent sequence. Think of it like putting together a jigsaw puzzle: Each piece may not make sense on its own, but together they create a complete moving image. Different neurons process different “puzzle pieces” of a real moving image, which the brain then assembles into a continuous scene.
The researchers also found that tadpole optic neurons can differentiate subtle changes in visual stimuli over time, capturing information in dynamic clips lasting about 100 to 600 milliseconds, rather than in static frames. These neurons respond well to patterns of light and shadow, and each neuron’s response to a specific part of the visual field helps to construct a detailed scene, a “movie fragment.” Kline and Hiramoto trained MovieNet to mimic this brain processing by encoding video clips as a series of small, identifiable visual cues. This allowed the AI model to spot subtle differences in dynamic scenes.
To test MovieNet, the researchers showed it video clips of tadpoles swimming in a variety of conditions. MovieNet achieved an impressive 82.3% accuracy in distinguishing between normal and abnormal swimming behavior, outperforming trained observers by about 18%. It even outperformed existing AI models, such as Google’s GoogLeNet, which achieved only 72% accuracy despite extensive training and resource usage. “This is where we saw significant promise,” says Klein.
The team found that MovieNet not only outperformed current AI models in understanding dynamic scenes, but it also required less data and processing time. MovieNet’s ability to simplify data without losing accuracy also sets it apart from traditional AI. By breaking down visual information into meaningful sequences, MovieNet effectively compresses the data like an archive file, while preserving important details. In addition to its high accuracy, MovieNet offers an environmentally friendly AI model. Traditional AI processing is energy-intensive, leaving a significant impact on the environment. MovieNet’s reduced data requirements provide a greener option that saves energy while maintaining high performance. “By mimicking the brain, we’ve created AI that is not only powerful but also robust,” says Klein. “This efficiency also opens up opportunities for AI to scale in industries where conventional approaches are expensive.”
MovieNet could have a significant impact on medicine as the technology continues to advance. It could be a valuable tool for detecting subtle changes in the early stages of conditions like irregular heart rhythms or early signs of neurodegenerative diseases like Parkinson’s. For example, small movement changes associated with Parkinson’s, which are often difficult to detect by the human eye, could be identified early on by AI, giving clinicians crucial time to intervene.
Additionally, MovieNet’s ability to detect changes in the swimming patterns of tadpoles exposed to chemicals could lead to more accurate drug screening methods, allowing scientists to study dynamic cell responses rather than relying on static images. “Current approaches miss critical changes because they only analyze time-lapse images,” Hiramoto notes. “Observing cells over time allows MovieNet to track the most subtle changes during drug testing.”