Quality assurance of voice systems: approaches to testing speech recognition devices

Introduction

The digital transformation of the economy and social sphere in the Russian Federation is accompanied by the active implementation of speech interfaces. Fr om voice assistants in mobile devices to voice robots in user support centers — automatic speech recognition and speech synthesis technologies are becoming the standard for citizen interaction with information systems.

Specifics of testing speech systems

A voice application is not a simple set of "microphone — processing" functions. It is a multi-level system, each element of which can cause failures:

1. Infrastructure level — quality of the received audio signal, time delays, operation of recognition and synthesis modules.

2. Application logic level — identification of user intent, accuracy of response generation, calling external functions.

3. Behavioral response level — correct handling of interruptions, pauses, changes in emotional tone.

4. Target result achievement level — completeness of the operation, user attention retention.

The key feature of speech systems is that a failure at any of the listed levels can ruin the user's impression of the service quality. For example, a flawlessly working linguistic model will be useless if the recognition module incorrectly transcribes the request due to background noise or pronunciation features.

According to industry research, about 30 percent of users express dissatisfaction with modern voice assistants, and the main reason cited is insufficient speech recognition accuracy. Moreover, systems that show high results on reference datasets may fail in real-world operating conditions due to audio stream compression, speech signal overlap, regional pronunciation variants, and language code switching.

Key metrics for speech recognition evaluation

The basis for testing speech recognition systems is objective quantitative metrics that allow measuring transcription accuracy.

Word Error Rate

Word Error Rate (hereinafter — WER) is a universally recognized reference criterion for evaluating recognition systems. This metric reflects the proportion of errors made by the system when recognizing words relative to the reference transcription. WER is calculated using the formula:

WER = (S + D + I) / N × 100%

wh ere:

- S — substitutions (incorrectly recognized lexemes);

- D — deletions (lexemes not recognized by the system);

- I — insertions (words added by the system that are absent in the original phrase);

- N — total number of words in the reference transcription.

Let us give an example. If the reference phrase is "I want to book a ticket to Moscow" and the system recognized "I want to book a ticket Moscow", then we have one substitution (of the preposition "to") and one deletion (of the preposition). WER = (1+1+0)/6 = 33.3 percent.

Limitations of Word Error Rate

Despite its widespread use, the WER metric has significant drawbacks that must be taken into account when interpreting results:

1. All errors are considered equal. Replacing the name "Ivan" with "Ivон" or with "Peter" represents the same error from the WER perspective, although in the first case the meaning is distorted only slightly.

2. Semantic significance is not taken into account. An error in a surname, phone number, or date is critical for service delivery, while skipping a filler word ("uh-uh") is acceptable and even desirable for natural perception.

3. Difficulties with processing speech disfluencies. If the user said "I need to uh-uh book a hotel" and the system recognized "I need to book a hotel", the WER metric will penalize the system for the deletion, although such a result is preferable.

To compensate for these limitations, it is recommended to use additional metrics such as Character Error Rate for evaluating recognition of proper names and surnames, as well as metrics based on the Jaro — Winkler distance.

Reference WER values for different conditions

It is important to understand that acceptable WER values depend on recording conditions and language specifics. For Russian in studio recording conditions, an excellent result is considered to be WER less than 5 percent, good — less than 8 percent, acceptable — less than 10 percent. In office noise conditions, threshold values shift: good result — less than 12 percent, acceptable — less than 15 percent.

It should be noted that Russian, with its developed morphology and free word order, imposes increased requirements on recognition systems. The phonetic similarity of some sounds (for example, sibilants and fricatives), as well as vowel reduction in unstressed positions, create additional difficulties.

Multi-level testing strategy

Experienced development teams test voice applications not as a single object, but step by step, by individual levels. This approach allows localizing the problem and determining at which stage the failure occurs.

Level 1. Testing speech scenarios and dialogue logic (automated, at the text level)

At this level, the "intellectual component" of the application is tested — the linguistic model and dialogue logic. All tests are conducted in text form, without connecting audio channels. This is the fastest and most economical testing method.

Examples of checks:

- Correctness of responses to typical user requests.

- Handling non-standard situations (for example, a request outside working hours).

- Protection against unauthorized interference with the logic of operation.

Level 2. Testing dialogue behavior (automated, at the text level)

At this stage, the system's ability to conduct multi-turn dialogue is tested: remembering the context of previous messages, correctly handling clarifying questions, maintaining the thread of conversation when changing topics. Text dialogue simulators are used.

Tooling: specialized libraries for automating browser tests and simulating user input.

Level 3. Testing the speech path (automated, with audio signal)

At this level, the operation of speech recognition and synthesis modules is tested in isolation. Pre-prepared audio recordings with reference transcriptions are used. WER, recognition accuracy in various acoustic conditions, and naturalness of synthesized speech are evaluated.

Level 4. End-to-end testing under real-world conditions (manual and automated)

The final stage is testing the entire system under conditions that maximally simulate real operation: with various accents, background noise, dialogue interruptions, and emotionally colored speech.

At this stage, the participation of a human tester becomes particularly important, since automatic quantitative metrics do not always reflect the real user perception. A user may forgive a slight response delay but will be annoyed by the unnatural or robotic sound of the virtual assistant's voice.

Testing multilingual systems

For applications operating in multiple languages, the testing procedure becomes significantly more complex. Simple mechanical translation of test scenarios does not ensure sufficient quality.

Key challenges in testing multilingual systems:

- Phonetic complexity. Languages with tonal systems (Chinese) or an abundance of guttural consonants (Arabic) require higher recognition accuracy.

- Lack of clear word boundaries. In Japanese, Chinese, and Korean, word segmentation is not a trivial task.

- Long compound lexemes. German and Finnish are known for compound words that may be absent from training samples.

- Language code switching. Users often mix languages within a single phrase, creating additional difficulties for recognition systems.

To test multilingual systems, it is necessary to develop equivalent scenarios, rather than literal translations. For example, the phrase "I need to make an appointment" should be rendered in Spanish not formally-grammatically, but taking into account the fixed speech patterns accepted in a particular region.

Tools for testing speech systems

Currently, a number of software products are available on the market that simplify the testing of voice solutions.

Vokal (TypeScript development environment)

The Vokal software package is an industrial solution for testing voice robots. The system supports interaction with cloud platforms for speech synthesis and recognition, and also allows overlaying background noise to create realistic test conditions.

Rehearse (Python development environment)

The Rehearse toolkit integrates with the pytest library and allows developing test scenarios for voice agents using familiar design patterns. It supports making real phone calls through gateways and semantic evaluation of responses using large language models.

Hamming AI

The Hamming AI platform offers a comprehensive solution for testing voice agents, including support for multilingual testing and continuous quality monitoring in production.

WildASR

A research testbed developed by Boson AI allows evaluating speech recognition systems in three directions: the influence of the acoustic environment, demographic characteristics of speakers, and linguistic diversity. A distinctive feature is the use of recordings of real human speech rather than synthesized samples.

Quality control in production

Testing procedures do not end with the application launch. Continuous monitoring of the system in real conditions is necessary for timely detection of quality degradation.

Key metrics for operational monitoring:

- Response time (latency). The value at the 50th percentile should be less than 1.5 seconds, at the 95th percentile — less than 3.5 seconds.

- Task completion rate — the percentage of sessions in which the user achieved their goal.

- Containment rate — the proportion of dialogues fully handled by the system without transfer to a human operator.

- WER by segments — tracking recognition quality for different accent groups, device types, and acoustic conditions.

Regular testing after each change to speech scenarios or model updates helps prevent quality degradation. Each failure detected during operation should be added to the regression test suite.

Conclusion

Quality testing of voice applications is a complex multi-level task that goes beyond testing individual components. A systematic approach is required, combining automated testing at various levels and manual investigation of user experience.

Basic principles for successful quality control organization:

1. Conduct testing step by step, by individual system levels, to localize error sources.

2. Use Word Error Rate as a baseline metric, supplementing it with other criteria to account for the semantic significance of errors.

3. Consider linguistic and cultural specifics when developing systems intended for multilingual use.

4. Simulate real operating conditions: acoustic noise, pronunciation variants, communication interruptions, unstable channel quality.

5. Implement continuous monitoring in production for timely detection of performance degradation.

Speech interfaces are becoming increasingly complex and context-dependent, and the quality of their testing directly determines whether users will trust the technology or become disillusioned with it. In light of the requirements of the Federal Law "On the State Language of the Russian Federation," correct processing of Russian-language speech in all its diversity is of particular importance. Investments in quality testing are investments in citizen trust and the success of digital service implementation.