Digital assistant showdown

Post_20150812_DigitalAssistants_Header

Apple revolutionized hands-free user interactions when they introduced Siri in 2011. Since then, we’ve seen the introduction of Google Now (2012), Hound from SoundHound, and most recently, a “leaked” copy of Microsoft’s Cortana for Android. Below is my biased and totally unscientific comparison of the current crop of personal assistant apps. Your results may vary.

Note: Hound is in private beta release and Microsoft Cortana is a “leaked” copy. Extra consideration should be given to them, considering they are not officially released products. Also, all of these products are backed by cloud services so they are continuously improving.

Contenders

Siri

Siri, the original personal assistant acquired by Apple in 2010, was the first voice-controlled user interface to utilize conversational speech and cloud intelligence. Continuing the Apple tradition of making technology approachable, Apple gave Siri a “personality”.

Distinguishing features:

  • Accessible via physical button on device, making hands-free operation easier.
  • Can learn your name and how to pronounce it correctly.
  • Accessible through the lock screen to help identify the owner of the device.
  • Only available on iPhone and iPad devices, although rumors of a OS X version have persisted for years.

Google Now

Personal assistant powered by Google’s industry leading search engine and natural language processing technology.

Distinguishing features:

  • Aggregates data using Google’s knowledge graph to predictively report when a variety of contextually relevant information on “cards”. Your orders are delivered, flight status, weather, traffic and limited support for 3rd party information.
  • First personal assistant to use always listening technology to trigger interactions using only the phrase “OK Google”
  • Available across platform as well as on the desktop, iOS, Android, OSX, Windows, Linux

Hound

Created by SoundHound and in development for over a decade, Hound touts mind boggling logic processing and speed. Hound also works amazingly well when using recommended queries, but becomes less useful when you depart from suggested questions. Hound is currently in private beta for Android only, but that will likely change as the digital personal assistant race heats up.

Distinguishing features:

  • Correctly interprets convoluted logic queries (e.g. “Show me all hotels in San Francisco for Friday night that are less than $200 dollars”).

Microsoft Cortana

Cortana originally debuted on Windows phones, and is now being ported to Android and, eventually, iOS. As I said earlier, I only tested the “leaked” version on Android, and will need to revisit once officially version has been released. Like Siri, Cortana attempts to have a personality, as well as some other novel features: geofencing, emoji story.

Distinguishing features:

  • Geofence aware reminders that send notifications based on location. So for example; “remind me to buy milk the next time I am at the store” and Cortana will remind you when you are geographically close to the store you select. I tried this and it works to an extent; I got reminders when i would drive past stores on the freeway.

Rules

Like all good competitions, I had to devise a point system to keep my decision as scientific as possible. I decided to award points as follows:

Speech recognition (10 points)

To test each assistant’s speech recognition, I ran it through a series of ten quick questions just to see if it even understood me. Each time the assistant responded, I gave it a point. If it didn’t, I gave it a goose egg.

Speed (4 points)

Speed is really subjective and dependent on a lot of factors outside of the app’s control (network speed, data signal, etc). Additionally some apps take longer because they are performing logical analysis of the query while others are just plugging data into a search engine. Since speed is so subjective, I graded on a curve and assigned a cumulative score based on each assistant’s overall speed performance.

Accuracy (5 points per query)

The true measure of a digital assistant is how well it answers your questions and requests. To test this, I asked each service the same ten questions and scored them accordingly. Below is a breakdown of how they were judged on a per-question basis:

0 = wrong answer or fail


1 = query was submitted to a search engine and the search results are displayed.


2 = answer is available but requires touch interaction


3 = answer or operation is correct and result is reported audibly


+1 = bonus point can be added for exceptional behavior


+1 = extra bonus if context is preserved on follow up questions. (e.g. Show me pictures of the Eiffel tower? How tall is it?)


-1 = points can be deducted for shady behavior (e.g. directing users to pay services)

Results

After a hard-fought software battle, here’s how each digital assistant netted out.

Event 1: Speech Recognition

Below are the 10 questions I asked each assistant:

“When was the pyramid of Cheops built?”


“What year did Arnold Schwarzenegger win Mr. Olympia?”


“How tall is the Matterhorn?”


“Give me directions to Hopdoddy?”


“Give me directions to Manchaca road?” ( Austin pronunciation )


“Give me directions to Manchaca road?” ( Spanish pronunciation )


“How do you pronounce Guadalupe?” ( Austin pronunciation )


“How do you pronounce Guadalupe?” ( Spanish pronunciation )


“What time is it in Tallahassee?”


“Who is the mayor of Tehran?”

And here are the results:

Siri


1 0 1 1 0 0 0 0 1 1 = 5




Google Now


1 1 1 1 1 1 0 1 1 1 = 9




Hound


0 1 1 0 0 0 0 1 1 1 = 5




Cortana


1 1 1 1 1 1 0 1 1 0 = 8

Event 2: Speed

Siri = 2


Google Now = 3


Hound = 4


Cortana = 1

Event 3: Accuracy

Since this is the most important event in the Digital Assistant Olympics, I’ll break down the results for each question/command individually.

1) “Play, ‘You know you like it.’”

Siri


Siri played the correct song from my music library, along with full media controls: Next, Skip, Previous, Pause, Stop, Resume.


Post_20150812_DigitalAssistants_sirimusic


Score = 5


(3 points for carrying out the command, 1 point for full control of music library, 1 point for context aware questions – “Who sings this song?” will report the artist for the currently playing track)

Google Now


Google Now played the correct song from my music library, but lacked the full media controls of Siri. All other actions must be performed by touch. In situations where the song or artist can not be identified in the music library, Google Now will launch Google Play Radio.


Post_20150812_DigitalAssistants_googmus


Score = 3


(3 points for carrying out the command)

Hound


Hound performed a web search and displayed the results from SoundHound instead of my music library. I also had to touch the screen to play the song.


Post_20150812_DigitalAssistants_houndmusic


Score = 2


(2 points for displaying the correct search results but requiring touch interaction to play)

Cortana


Cortana performed a web search, resulting in series of YouTube videos I had to click to activate. However, it should be noted that Cortana documentation says this works appropriately on Windows phones.


Post_20150812_DigitalAssistants_cortmusic


Score = 1


(1 point for performing a web search that led me to a bunch of videos instead of the song itself)

SUMMARY:

Siri = 5


Google Now = 3


Hound = 2


Cortana = 1

2) “What time does Torchy’s close? Show me their menu.”

Siri


Siri successfully answered the question, but required additional effort to dig up the menu.


Post_20150812_DigitalAssistants_siritorch


Score = 3


(3 points for answering the question properly.

Google Now

Google Now answered the question and brought up the menu when asked.


Post_20150812_DigitalAssistants_googtorch


Score = 5


(3 points for answering the question, +1 for reporting the results without follow up questions, +1 for bringing up their menu upon voice command.)

Hound


Hound performed a web search, but didn’t even narrow it down according to my location.


Post_20150812_DigitalAssistants_houndtorch


Score = 1


(1 point for performing a web search)

Cortana


Cortana answered the question, as well as the audible follow-ups, but I still had to tap for menu.


Post_20150812_DigitalAssistants_corttorch


Score = 4


(3 points for answering the question, +1 for reporting the results without follow up questions)

SUMMARY:

Siri = 3


Google Now = 5


Hound = 1


Cortana = 4

3) “Make dinner reservations for tomorrow night.”

Siri


Siri worked due to the OpenTable integration, however, I had to use touch screen to answer follow up questions.


Score = 4


(3 points answering command, +1 for seamless OpenTable integration)

Google Now


Google Now performed a google search


Score = 1


(1 point for performing a search)

Hound


Hound performed a search


Score = 1


(1 point for performing a search)

Cortana


Cortana displayed list of nearby restaurants according to my geographical location.


Score = 2


(1 point for performing a search, +1 point for using my location to tailor the search results)

SUMMARY:

Siri = 4


Google Now = 1


Hound = 1


Cortana = 2

4) “Wake me up at 10:00 AM.”

Siri


Siri created an alarm and allowed me to verbally cancel or adjust the time.


Score = 4


(3 points for performing the task, +1 for being able to adjust times)

Google Now


Google Now successfully created the alarm, but nothing else.


Score = 3


(3 points for performing the task)

Hound


Hound successfully created the alarm, but nothing else.


Score = 3


(3 points for performing the task)

Cortana


Cortana successfully created the alarm, but nothing else.


Score = 3


(3 points for performing the task)

SUMMARY:

Siri = 4


Google Now = 3


Hound = 3


Cortana = 3

5) “Remind me to {do something} at {time}.”

Siri


Siri created the reminder, but it seemed to be stuck in a loop and required several attempts


Score = 2


(3 points for performing the task, -1 for being buggy)

Google Now


Google Now created the reminder.


Score = 3


(3 points for performing the task)

Hound


Hound created the reminder.


Score = 3


(3 points for performing the task)

Cortana


Cortana created the reminder and added a location to it.


Score = 4


(3 points for performing the task, +1 for geofencing)

SUMMARY

Siri = 2


Google Now = 3


Hound = 3


Cortana = 4

6) “Is it safe to take acetaminophen with alcohol? What about aspirin?”

Siri


Siri performed a web search.


Score = 1


(1 point for performing a search)

Google Now


Google Now audibly returned the correct result and answered the follow up question.


Score = 4


(3 points for performing request, +1 for maintaining context during follow-up)

Hound


Hound performed a web search.


Score = 1


(1 point for performing a search)

Cortana


Cortana performed a web search.


Score = 1


(1 point for performing a search)

SUMMARY:

Siri = 1


Google Now = 4


Hound = 1


Cortana = 1

7) “Who was president of the United States in 1993? How long was he president?”

Siri


Siri audibly answered Bill Clinton, but failed to acknowledge the end of George Bush’s term.


Score = 2


(3 points for performing request, -1 for leaving out a president)

Google Now


Google Now audibly mentioned both presidents.


Score = 3


(3 points for performing request)

Hound


Hound performed a web search.


Score = 1


(1 point for performing a search)

Cortana


Cortana performed a web search.


Score = 1


(1 point for performing a search)

SUMMARY:

Siri = 2


Google Now = 3


Hound = 1


Cortana = 1

8) “Who was president of the Germany in 1987

Siri


Siri audibly returned the correct results – Richard Von Weizsacker


Score = 3


(3 points for performing request)

Google Now


Google Now audibly reported a completely unrelated anecdote about President Reagan’s “Tear down this wall” speech. (I asked this question on a historically significant day, which might have confused Google.)


Score = 0


(+1 point for answering audibly with confidence, -1 for giving the wrong answer)

Hound


Hound performed a web search.


Score = 1


(1 point for performing a search)

Cortana


Cortana performed a web search.


Score = 1


(1 point for performing a search)

SUMMARY:

Siri = 3


Google Now = 0


Hound = 1


Cortana = 1

9) “What is a calico cat? Why are they always female?”

Siri


Siri performed a web search.


Score = 1


(1 point for performing a search)

Google Now


Google Now audibly returned the correct results sourced from Wikipedia.


Score = 4


(3 points for performing request, +1 for preserving context during follow-up)

Hound


Hound performed a web search.


Score = 1


(1 point for performing a search)

Cortana


Cortana returned the correct results sourced from Wikipedia, but not audibly


Score = 2


(3 points for performing request, -1 staying quiet)

SUMMARY:

Siri = 1


Google Now = 4


Hound = 1


Cortana = 2

10) “Find a hotel in San Francisco for Friday night that is less than $200, and has a free breakfast”

Siri


Siri performed a web search for San Francisco the city, saying nothing about the hotels.


Score = 0


(0 points for performing the wrong search)

Google Now


Google Now performed a web search for San Francisco hotels.


Score = 1


(1 point for performing a search)

Hound


Hound returned correct results with prices and links to book the hotel. This is one of the featured query types that Hound excels at.


Score = 5


(3 points for performing an accurate search, +1 for providing the right links, +1 for speed)

Cortana


Cortana performed a web search for San Francisco hotels.


Score = 1


(1 point for performing a search)

SUMMARY:

Siri = 0


Google Now = 1


Hound = 5


Cortana = 1

Winners

Once the checkered flag dropped, it was Google Now who put the pedal to the metal and crossed the finish line just in time to gain the gold medal. Between its unrivaled contextual awareness and search prowess, Google Now was just a little too much for its competitors. Siri wasn’t far behind, and could become a true contender in the near future. Coming in third was Cortana, which may perform even better once it is officially released. And last, and unfortunately least, was Hound. Hound put up a good fight, but it’s just too small to truly compete with superpowers like Google, Apple, and Microsoft. If you don’t feel like adding up the scores yourself, I’ve added the final tallies below. As the old adage goes, numbers don’t lie.

Siri = 32


Google Now = 39


Hound = 28


Cortana = 29

Max Score (including bonus points): 64

Related Articles

Stay-Up-To-Date

Keep in the loop with the latest in emerging technology and Mutual Mobile