Teaching software to predict handshakes, hugs and kisses

It sounds like the premise of a science fiction movie, but researchers at MIT's Computer Science and Artificial Intelligence Laboratoryare teaching computers to see into the future.

They've created an algorithm that can forecast hugs, kisses, and high-fivesbefore they even happen.

CBC Radio technology columnist Dan Misenerexplainshow computers are getting better and better at understanding and anticipating human behaviour.

What have these MIT researchers created?

They've developed an algorithm that can look at a photo of two peopleand predict what's going to happen next.

For instance, if I showed this software a photo of you and memeeting on the street, it can anticipate whether we're likely to hug, kiss, shake handsor high-five.

This is the sort of thing humans are pretty good at. Based on a lifetime of watching people interact, we can read social situations and body language and make educated guesses about how two people are likely to interact.

But of course, computers don't have a lifetime of experience, and they don't have common sense.That's the challenge the researchers are trying to tackle.

• Microsoft's new emotion-detecting AI can be hilariously bad at detecting emotions

They're tackling it using artificial intelligence, deep learning, and computer visionall significant trends in technology right now.

• Deep Learning Godfather says machines learn like toddlers

• Computer shows human-like learning for writing characters

• Facial recognition tech is allowing stores to reward customers

How did they teach the algorithm to predict hugs?

If you want to train a computer to recognize human behaviour, you need to feed it lots of examples.

Soaccording to MIT researcher Carl Vondrick,they turned to a massive collection of recorded human behaviourYouTube.

"We had downloaded 600 hours of video from YouTube,"he said.

And the algorithm learned about human behaviour by watching YouTube videos.Some of the videos were clips from TV shows like Desperate Housewivesand The Office. Others were user-generated.And with each clipthe algorithm watches, it learns something about human behaviour.

"So it's learning, for example, that when someone's hand is outstretched, that means a handshake is going to come," Vondrick said."Or maybe that [if] two heads are very close together, that mightsuggest a kiss."

Once the researchers trained the algorithm, it could make predictions or forecasts a few seconds in the future.So they could pause a video, and ask the algorithmwhat do you think is going to happen next? Will they shake hands?Will they kiss? Will they hug? Or will they high-five?

How accurate is it?

Vondrick said right now, the system is about 43 per centaccurate at predicting one of those four actions.That's a lot better than the one-in-four odds you'd have if you simply guessed randomly.

But it's not nearly as good as human beings. Again, humans have intuition, and we're pretty good at picking up on context, body languageand all sorts of things that aren't captured in video.

That said, human intuition isn't always right. For instance, there are those moments where one person offers a handshake, but the other person leans in for a hug. And both people feel slightly awkward. So even humans aren't perfect at predicting the future.

Vondrick thinks his system can get much more accurate, and the key will be exposing it to more video. They're using machine learning techniques, so the system should get better the more training data it has.

• Robot learns to cook pizza by watching videos on the internet

Whatcould this algorithm be used for?

Predicting high-fives is interesting, but practically speaking, it's not all that useful.

But Vondrick saidwith enough training data, the algorithm could be used to predict any kind of human behaviour.

He said he's particularly excited about potential health care applications:"So if we can predict that someone's going tofall, or maybe hurt themselves, or maybe start a fire, maybe we can alert emergency responders, or even try to prevent the accident from happening."

This is possible, because sometimes there are indicators that show up long before you actually fall.

• Spark: Using theMicrosoft Kinectas an early-warning system

And if technology can help monitor for those signs,that could be a real win.

• VIDEO: Companies racing to develop artificial intelligence