Home AI Market ResearchEnabling Technology & Infrastructure Voice Assistant Benchmark 2.0 (2019)

Voice Assistant Benchmark 2.0 (2019)

by rschmelzer

Document ID: CGR-VAB19 | Last Updated: August 31, 2019

 

Report Overview
Abstract:

Voice assistants are voice-based conversational interfaces paired with intelligent cloud-based back-ends. The device itself provides basic Natural Language Processing (NLP) and Natural Language Generation (NLG) capabilities, and the back-end intelligence gives these devices AI-powered intelligence. Examples of voice assistants include Amazon Alexa, Apple Siri, Google Home, and Microsoft Cortana.

Cognilytica is focused on the application of AI to the practical needs of businesses, and because we believe voice assistants can be useful to those businesses. As such, we need to understand the current state of the voice assistant market. We care about what happens when the NLP does its processing and those outputs are provided as input to an intelligent back-end system. Some vendors are demoing their devices being used to make real-world calls to real-world businesses to perform real-world tasks. This is not just playing music or telling you the weather. This requires at least a minimum level of intelligence to perform without frustrating the user. And so, without an understanding of what the limitations of these devices intelligence really are, we’re left wondering what sort of applications these voice assistants are best suited for.

In this report, Cognilytica evaluates the intelligence and knowledge graph capabilities of four voice assistants: Amazon Alexa, Google Assistant (Home), Apple Siri, and Microsoft Cortana. We want to know — just how intelligent is the AI back-end?

Key Findings:

  • The Voice Assistant Benchmark determines the underlying intelligence of voice assistant platforms, and identifies categories of conversations and interactions to determine intelligence capabilities
  • Voice assistants have a long way to go before even half of the responses are acceptable.
  • For the current benchmark, Alexa provided the most number of adequate responses, with 34.7% of responses determined to be adequate while Google followed close behind with 34.0% adequate. Cortana showed much improvement from the previous benchmark with 31.9% adequate responses while Apple’s Siri still trails with 24.3% of answers determined to be adequate.

Key Vendors Included in this Report:

  • Amazon Alexa
  • Google Assistant (Google Home)
  • Apple Siri
  • Microsoft Cortana

Report Details:

  • 52 Pages
  • 28 Charts
  • 15 Tables
Icon

Voice Assistant Benchmark 2.0 (2019) [CGR-VAB19] 1.64 MB

Voice assistants are voice-based conversational interfaces paired with intelligent...

Price: $995

Cognilytica Access Subscribers get free access to this report!
Table of Contents
  • Executive Summary    5
    • Key Findings    5
    • Benchmark Details    6
  • About the Voice Assistant Benchmark    6
    • Testing Cloud-based Conversational Intelligence Capabilities of Edge Voice Assistants    6
    • What this Benchmark Aims to Test:    8
    • What this Benchmark Does NOT Test:    8
    • Yes, We Know Voice Assistants Aren’t Smart… But the Bar is Moving.    8
    • Purpose of Benchmark: Measure the Current State of Intelligence in Voice Assistants    9
    • If you’re building Voice-based Skills or Capabilities on Voice Assistant Platforms, you NEED to Pay Attention    9
    • This is Not a Ranking!    9
  • Benchmark Methodology    9
    • Open, Verifiable, Transparent. Your Input Needed.    10
    • Benchmark Configuration    11
    • Voice Assistants Tested:    11
    • Computer Generated Voice(s) Used:    11
  • Voice Assistant Benchmark 1.0 Questions    12
    • Benchmark Calibration Questions (CQ)    12
      • Overview:    12
      • Current Benchmark Questions:    12
    • Concept Understanding (CU) Benchmark Questions    13
      • Overview:    13
      • Current Benchmark Questions:    13
    • Understanding Comparisons (UC) Benchmark Questions    14
      • Overview:    14
      • Current Benchmark Questions:    14
    • Understanding Cause & Effect (CE) Benchmark Questions    15
      • Overview:    15
      • Current Benchmark Questions:    15
    • Reasoning & Logic (RE) Benchmark Questions    16
      • Overview:    16
      • Current Benchmark Questions:    16
    • Helpfulness Benchmark (HP) Questions    17
      • Overview:    17
      • Current Benchmark Questions:    17
    • Emotional IQ (EI) Benchmark Questions    18
      • Overview:    18
      • Current Benchmark Questions:    18
    • Intuition and Common Sense (IN) Benchmark Questions    19
      • Overview:    19
      • Current Benchmark Questions:    19
    • Winograd Schema Inspired (WS) Benchmark Questions    20
      • Overview:    20
      • Current Benchmark Questions:    20
    • Slang / Colloquialisms / Expressions (SE) Benchmark Questions    21
      • Overview:    21
      • Current Benchmark Questions:    21
    • Miscellaneous Questions    22
      • Overview:    22
      • Current Benchmark Questions:    22
    • Deductive & Probabilistic Reasoning (DR) Benchmark Questions    23
      • Overview:    23
      • Current Benchmark Questions:    23
    • Entity Resolution (ER) Benchmark Questions    24
      • Overview:    24
      • Current Benchmark Questions:    24
  • Benchmark Results: Calibration Questions    25
    • Overview:    25
    • Complete Results    25
    • Analysis of Results    26
  • Benchmark Results: Understanding Concepts Questions    27
    • Overview:    27
    • Complete Results    27
    • Analysis of Results    28
  • Benchmark Results: Understanding Comparisons    29
    • Overview:    29
    • Complete Results    29
    • Analysis of Results    30
  • Benchmark Results: Understanding Cause & Effect    31
    • Overview    31
    • Complete Results    31
    • Analysis of Results    32
  • Benchmark Results: Reasoning & Logic    33
    • Overview:    33
    • Complete Results    33
    • Analysis of Results    34
  • Benchmark Results: Helpfulness Questions    35
    • Overview    35
    • Complete Results    35
    • Analysis of Results    36
  • Benchmark Results: Emotional IQ Questions    37
    • Overview    37
    • Complete Results    37
    • Analysis of Results    38
  • Benchmark Results: Intuition and Common Sense    39
    • Overview    39
    • Complete Results    39
    • Analysis of Results    40
  • Benchmark Results: Winograd Schema Inspired    41
    • Overview    41
    • Complete Results    41
    • Analysis of Results    42
  • Benchmark Results: Slang / Colloquialisms / Expressions    43
    • Overview    43
    • Complete Results    43
    • Analysis of Results    44
  • Benchmark Results: Miscellaneous Questions    45
    • Overview    45
    • Complete Results    45
    • Analysis of Results    46
  • Benchmark Results: Deductive Reasoning Questions    47
    • Overview:    47
    • Complete Results    47
    • Analysis of Results    48
  • Benchmark Results: Entity Resolution Questions    49
    • Overview:    49
    • Complete Results    49
    • Analysis of Results    50
  • Total Results & Overall Analysis    51
  • Research Method and Statement of Opinion    51
  • Related Research    52
  • About Cognilytica    52
Icon

Voice Assistant Benchmark 2.0 (2019) [CGR-VAB19] 1.64 MB

Voice assistants are voice-based conversational interfaces paired with intelligent...
Video Highlights
https://www.youtube.com/watch?v=eqKKyqQzNGc

 

Statement of Opinion & Terms and Conditions of Sale
Although Cognilytica believes that the results, conclusions, and analysis produced in support of this report are well informed, comprehensive, and reasonable, Cognilytica cannot guarantee future results, accuracy of market predictions, or applicability of conclusions to report purchaser or reader’s business. Moreover, Cognilytica does not assume responsibility for the accuracy and completeness of such statements. The information derived in this report are statements of opinion only, and Cognilytica shall not be held liable in any manner for any conclusions or actions taken pursuant to this report. The information contained herein has been obtained from sources believed to be reliable. Cognilytica shall have no liability for errors, omissions, or inadequacies in the information contained herein or for interpretations thereof. Report purchaser and/or reader assumes sole responsibility for the selection of these materials to achieve its intended results. The opinions expressed herein are subject to change without notice. Cognilytica does not make open its research methods, underlying data, sources, or means and methods of analysis for inquiry, evaluation, or examination.

 

Related Content

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept