When trying to learn a physical skill such as judo or yoga from a video, it can be frustrating if you're not able to keep up with the onscreen action. That's where Reactive Video playback comes in, as its slows videos down to match the speed of their viewers.
The system is being developed via a collaboration between Britain's Lancaster University, California's Stanford University, and the FX Palo Alto Laboratory research center.
Hardware-wise, its main component is a Microsoft Kinect motion-sensing module. Utilizing the output from that device, special software continuously tracks the positions of the viewer's elbows, knees, hands, arms, hips and legs. This data is used to create a stickperson-like real-time animated model of them.
That same "skeleton-tracking" software is used to analyze the video before the session starts, creating a similar animated model of the onscreen instructor. Once the session gets underway, the system compares the recorded movements of the instructor's model to the corresponding real-time movements of the viewer's model.
As the instructor is performing a certain action, probabilistic algorithms determine approximately how long it will take the viewer to do that same thing. The video is then automatically slowed down, in order to keep the instructor from getting too far ahead of the viewer.
And as an added bonus, the technology can be used on existing videos that haven't been shot or edited in any specific fashion.
"For some people, keeping pace can be tricky – especially when learning something new, and for older people or those with impaired movement," says Lancaster's Dr. Christopher Clarke, co-author of a paper on the research. "Also, constantly reaching for a remote to pause, rewind and replay, can be frustrating and breaks the immersion. Our system overcomes these issues by having the video automatically adjust itself to play back at the user’s speed, which is less stressful and more beneficial for learning."
The paper was recently published in the journal Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology.
Reactive Video is demonstrated in the clip below.
Source: Lancaster University