Skip to main content

8 Children Test Their Speech on Siri, Echo and Google Home

Andy Wood and Matt Kirshen test the limits of everyday AI by using children of a variety of ages in linguistics tests designed to determine which AI is the best at understanding the most people. Featuring Google Home, Amazon Echo, and Siri. Check out more of Matt and Andy on their podcast at http://www.probablyscience.com

Released on 11/09/2017

Transcript

Hey, I'm Matt Kirshen.

And I'm Andy Wood.

And we wanted to know which of these different AIs

are best able to handle different ages of children.

Where do babies come from?

[Alexa] I didn't find Baby Come Close

single version in your library.

Where's babies come from?

[Cortana] The birth canal opens for--

Okay, okay, Cortana! Okay, Cortana! Good job.

So we've got Amazon Echo, Google Home,

and running on an iPhone we have both Siri and Cortana.

And we're gonna try out all four of these devices

with eight children whose ages range

from two to just under six.

So we'll see at what age these various devices

start to be able to understand children's speech.

(funky jazz)

What does horse say?

(buzzer)

[Alexa] What do you want to play?

That rhymes!

It's getting close. It was close.

What did the (mumbles)?

(buzzer)

[Alexa] Hmm, I don't know that one.

Okay, Google.

What does a horse say?

[Google Home] Here's what a horse sounds like.

(ding) (horse whinny)

What does horse say?

(buzzer)

What does a horse say?

[Google Home] Here's what a horse sounds like.

(ding) (horse whinny)

Here's what a horse sounds like.

(ding) (ding)

(horse whinny) (ding)

[Male Speaker] Nice job!

[Female Speaker] You're going to say,

What does the horse say?

(buzzer)

What does a horse sing?

[Siri] A horse sounds like this.

(ding) (horse whinny)

What does a horse say?

[Female Speaker] It brought up a horse.

(ding)

What does a horse say?

[Cortana] Whinny. (ding)

(group cheers and claps)

(babbling)

(buzzer)

What does a horse say?

[Cortana] (ding) Whinny.

Whinny?

The winner of this round is a tie: Google and Cortana.

And who knew that a horse says, Whinny.

Apparently a horse says the word whinny.

(upbeat techno)

[Andy] Do you want to say, Where do babies come from?

Yes!

Alexa, where do babies come from?

[Alexa] I think they are delivered to families by storks.

(ding)

Where do babies come from?

[Alexa] I didn't find Baby Come Close single version

in your library. (buzzer)

Where do babies come from?

[Alexa] Sorry, I'm not sure about that.

Babies?

[Alexa] The Beatles were an English rock band--

(quiet laughter) What beetle?

[Alexa] formed in Liverpool.

What beetle?

Where do babies come from?

[Alexa] I think they are delivered to families by storks.

(ding)

Okay, Googles, where does babies come from?

[Google Home] Well, babies come from seeds, too.

[Matt and Andy] Okay, Google, stop.

Where does the baby come from?

[Google Home] When the baby starts to crown--

Okay. (laughs)

(Google Home is drowned out) (ding)

Okay, Google.

That's impressive and very detailed.

Where do babies come from?

[Siri] I found this on the web

for Where do babies come from.

(ding)

Okay.

(babbling) come from?

[Siri] The daddy has the seed--

(ding)

and the mommy has the egg.

Where's babies come from?

[Cortana] The birth canal opens--

Okay, okay, Cortana! Okay, Cortana! Good job.

Where do babies come from?

[Cortana] I called up some videos for you.

Oh no!

Okay!

[Female Speaker] Hush, Little Baby

(buzzer)

[Male Speaker] Oh, okay, okay.

It was another tie.

This time, Google and Siri.

So, some AIs went with a sort of cutesy stork answer,

which is a little bit more discreet

but medically inaccurate.

Other devices went very medically accurate

Too medically accurate.

Excessively medically accurate.

(thumping bass)

Just a difficult one-- Yeah.

like, who is Benedict Cumberbatch?

No!

(buzzer)

What did Benny come batch?

[Alexa] Hmm, I don't know that.

(buzzer)

Who is Benedict Cumberbatch?

[Alexa] Benedict Cumberbatch--

(ding)

is an English actor-- Very good.

who has performed.

Can the froggy do it?

(babbling)

(buzzer)

Do you want to try saying Benedict Cumberbatch?

Yeah!

Okay, Google.

(buzzer)

Who is Ben bid catch?

Benedict cabbage?

Benedict Cumberbatch?

(ding)

[Google Home] Benedict Cumberbatch is an English actor.

(cheering and laughter)

Who is Benedict Hubbobatch?

Hummlebit Pubberbatch?

Penedict Cumberpatch?

[Google Home] Benedict Cumberbatch

(ding)

[Male Speaker] You got it!

Who is Imogen Cumberbatch?

[Google Home] Benedict Cumberbatch is

(ding)

(laughter)

Who's Benedict Cumberbatch?

[Female Speaker] Hold up.

Who is (buzzer)

Benedict to Cumberbatch?

Who is Benedict Cubberpatch?

[Siri] Here's some information.

(ding)

[Male Speaker] No, it worked!

Who is Benedict Popperbatch?

[Cortana] Benedict Cumberbatch

(ding)

(group laughs)

Who is Bindenson Cumberbatch?

[Cortana] Bing is my favorite search engine.

(buzzer)

She is not getting this right.

[Andy] Google Home knocked this one outta the park.

It is not easy for children to say Benedict Cumberbatch

but when they try, it is a delight.

(staticky bass music)

Where is tati go?

[Alexa] (babbling) Should I add

the station for Talk to God to your Pandora account?

(buzzer)

Where is Ardantica?

[Alexa] Sorry, I don't know that.

(buzzer)

Where is Ardonticats?

[Alexa] Sorry, I'm not sure.

Where is at-terdider?

(buzzer)

(laughs)

Where's Antarntica?

[Alexa] Antarctica is traditional continent

in the southern hemisphere.

(child gasps)

(group cheers)

(babbling)

(buzzer)

Okay, Google, where is Ardanca?

[Google Home] Arnica, Montana is made up of--

(buzzer)

Where's Atopterca?

[Google Home] Sorry, I don't understand.

She didn't know where that is.

Where is Antofterca?

[Google Home] I heard, Where is Antarctica?

(ding)

I think she knows where Antarctica is.

Okay, Google, where's Antarctica?

[Google Home] Here is some information about Antarctica.

(ding)

Where's Inartica?

[Siri] Here are the top search results.

Arctica ski race clothing.

(buzzer)

That's pretty close.

Where is Antonfrica?

[Siri] I couldn't find any matching places.

(buzzer)

Where's Antontica?

[Siri] Sorry, I couldn't find Sir Antic

(buzzer) in your contacts.

Where's Antarntica?

[Cortana] Antarctica is--

(ding)

Yeah!

Yeah!

[Andy] It was a tie once again

between Google Home and Cortana.

[Matt] So, this one was the lowest scoring round.

And in this one, it was less a test of the AI

and more a test of these children's ability

to say a difficult word.

(hip-hop music)

Are you a Publicate or a double cup?

[Alexa] Sorry.

(buzzer) I'm not sure about that.

A Democrat or Republican?

[Alexa] Sorry. (buzzer)

I'm not sure about that.

I think that means she's not one.

Are you a Democrat or a Republican?

[Google Home] I like all parties.

(ding)

Are you Republican or a Democrat?

[Google Home] Sorry, I don't understand.

I can do this.

I can do this!

Are you a Democrat or Republican?

[Google Home] Yes.

(buzzer)

I think that means both.

Are you a Democrat or a Republican?

[Female Speaker] It picked up, Are you a Democrat

or Republican? (buzzer)

Well, that's stupid of Siri.

Are you a devil head or Republican?

[Siri] Hmm, that's something I don't know.

(buzzer)

Are you Antican or (mumbles)?

[Siri] I don't get it but I can check the web for,

I am answer kin.

Are you a Democrat or Repubwiquin?

[Siri] We were talking about you.

(buzzer)

Are you a Democrat or a Republican?

[Cortana] We're not allowed to register as voters.

(ding)

Are you a double cup or a Republican?

[Female Speaker] It registered as, Are you double crap

(buzzer) or a Republican?

[Andy] On this one, Cortana scored the best.

Here's one where context would've really helped

and these devices mostly weren't able to put it together.

All right, results time.

So, different devices did better with different phrases.

Mm-hmm.

Siri tends to present web pages as answers

to the questions, rather than just answering the question.

Uh, it's reds.

Here's the thing I noticed with the five year olds,

is they started to ask their own questions.

[Andy] Is there anything you wanted to ask?

Yes.

Alexa, who is Cottontail?

How does a police car get a bad guy?

Sing me a song, Google.

(excited noises) I got it!

Wait! Wait!

Okay, Google, how do I get a job?

Pretty much every device all Gia wanted to do was find out

how to get a boyfriend.

How do I get a boyfriend?

Okay, Google, how do I get a boyfriend?

So, Andy, the leaderboard.

Tied for third place overall, we had Alexa and Siri.

Second place, Cortana.

Nearly did it, right?

Cortana came joint first in a few of the rounds.

Yup but it wasn't enough to beat the champion

from the last video.

Winner again.

[Andy] Google Home.

[Matt] Cumberbatch just tipped you over the edge.

Yeah. (laughs)

So, if you are Pemberton Cummingbatch--

And you ever find yourself in Anoptica--

And you're a double craps.

Or a Publican.

Let us know! Comment underneath.

Starring: Matt Kirshen, Andy Wood