AMRNB and AMRWB codecs as described by IETF

The Adaptive Multi-Rate speech codec

The AMR codec is a multi-mode codec with 8 narrow band speech modes with bit rates between 4.75 and 12.2 kbps. The sampling frequency is 8000 Hz and processing is done on 20 ms frames, i.e. 160 samples per frame. The AMR modes are closely related to each other and use the same coding framework

Below three of the AMR modes are already adopted

PDC-EFR : 6.7Kbps
IS-641 : 6.7 Kbps
GSM-EFR : 12.2Kbps

The Adaptive Multi-Rate Wideband speech codec

The AMR-WB codec is a multi-mode speech codec with 9 wideband speech  coding modes with bit-rates between 6.6 and 23.85 kbps. The sampling frequency is 16000 Hz and processing is performed on 20 ms frames, i.e. 320 speech samples per frame. The AMR-WB modes are closely related to each other and employ the same coding framework.

Common Characteristics for AMR and AMR-WB

The multi-mode feature is used to preserve high speech quality under a wide range of transmission conditions. In mobile radio systems (e.g. GSM) mode adaptation allows the system to adapt the balance  between speech coding and error protection to enable best possible speech quality in prevailing transmission conditions. Mode adaptation  can also be utilized to adapt to the varying available transmission bandwidth. Every codec implementation MUST support all specified speech coding modes.  The codecs can handle mode switching to any mode at any time, but some transport systems have limitations in the  number of supported modes and on how often the mode can change. The  mode information must therefore be transmitted together with the speech encoded bits, to indicate the mode. To realize rate adaptation  the decoder needs to signal the mode it prefers to receive to the  encoder. It is RECOMMENDED that the encoder follows a received mode request, but if the encoder has reason for not follow the mode  request, e.g. congestion control, it may use another mode. No codec mode request MUST be sent for packets sent to a multicast group, and the encoder in the sender SHOULD ignore mode requests when sending to  a multicast session but MAY use RTCP feedback information as a hint that a mode change is needed.

Both codecs include voice activity detection (VAD) and generation of comfort noise (CN) parameters during silence periods. Hence, the  codecs have the option to reduce the number of transmitted bits and  packets during silence periods to a minimum. The operation to send CN parameters at regular intervals during silence periods is usually called discontinuous transmission (DTX) or source controlled rate (SCR) operation. The frames containing CN parameters are called Silence Indicator (SID) frames.

references:
https://www.ietf.org/proceedings/51/I-D/draft-ietf-avt-rtp-amr-10.txt

Codec Latency vs. Bandwidth Optimization

The low-bandwidth codecs are quite efficient. For example, G.729 will compress 10 milliseconds of audio to 10 bytes and G.723.1 encodes 30ms frames to 24 or 20 bytes.

However, since we send compressed audio frames as payload in RTP packets which are in turn sent over UDP, we need to consider the overhead for IP, UDP, and RTP headers. The overhead is 40 bytes per packet. This is significant when compared with the size of a compressed audio frame if we are not on a local area network and the bandwidth is limited. The table below shows the overhead for several low-bandwidth codecs.

If you want to improve bandwidth utilization, the obvious way to go is to send more frames in one RTP packet. However, as you do this, you also increase latency. If you decide to send, say, 100 milliseconds of audio in one packet. this means that you have added a latency of the same 100 milliseconds. Simply put, the first sample of the first frame arrives together with the last sample of the last frame in the packet, so the total delay is equal to the length of audio carried in the packet.

There is a recommendation that round-trip latency should not exceed approximately 300 milliseconds, otherwise people will start noticing.

When calculating the latency, you need to consider the time it takes to send a packet from one end to the another (your mileage may vary, try to use "traceroute" to get a clue) and the size of the jitter buffer of the receiving end (which can be 50-60 milliseconds worth of audio). Considering all this, I would say the reasonable maximum is to send 60 milliseconds of audio in one packet. This will result in the following bitrates:

In addition to latency, there are two more things you should consider when increasing the number of audio frames per RTP packet:

If a packet with a larger number of frames gets lost, the loss is more noticeable to the user.
With greater end-to-end delay, possible echos become more noticeable.

references:
http://toncar.cz/Tutorials/VoIP/VoIP_Basics_Codec_Latency_vs_Bandwidth.html

Overview of Voice Codecs

Once we have the audio signal represented as a sequence of samples, the next step is to compress it to reduce the consumption of network bandwidth required to transmit the speech to the receiving party.

The compression and decompression is handled by special algorithms we call codecs (COder-DECoder)

All the codecs listed below here expect the input to be audio sampled at 8 kHz with 16-bit samples.

G.711
G.711 is a codec that was introduced by ITU in 1972 for use in digital telephony, i.e. in ISDN, T.1 and E.1 links. The codec has two variants: A-Law is being used in Europe and in international telephone links, u-Law is used in the U.S.A. and Japan.

G.711 uses a logarithmic compression. It squeezes each 16-bit sample to 8 bits, thus it achieves a compression ratio of 1:2. The resulting bitrate is 64 kbit/s for one direction, so a call consumes 128 kbit/s (plus some overhead for packet headers). This is quite a lot when compared with other codecs.

This codec can be used freely in VoIP applications as there are no licensing fees. It works best in local area networks where we have a lot of bandwidth available. It's benefits include simple implementation which does not need much CPU power (can be implemented using a relatively simple table lookup) and a very good perceived audio quality - the MOS value is 4.2.

G.729

G.729 is a codec that has low bandwidth requirements but provides good audio quality (MOS = 4.0). The codec encodes audio in frames, each frame is 10 milliseconds long. Given the sampling frequency of 8 kHz, the 10 ms frame contains 80 audio samples. The codec algorithm encodes each frame to 10 bytes, so the resulting bitrate is 8 kbit/s for one direction.

When used in VoIP, we usually send 3-6 G.729 frames in each packet. We do this because the overhead of packet headers (IP, UDP, and RTP together) is 40 bytes and we want to improve the ratio of "useful" information.


G.729 is a licensed codec. As far as end users are concerned, the easiest path to using it is to buy a hardware that implements it (be it a VoIP phone or gateway). In such case, the licensing fee has already been paid by the producer of the chip used in the device.

A frequently used variant of G.729 is G.729a. It is wire-compatible with the original codec but has lower CPU requirements.


G.723.1

G.723.1 is a result of a competition that ITU announced with the aim to design a codec that would allow calls over 28.8 and 33 kbit/s modem links. There were two very good solutions and ITU decided to use them both. Because of that, we have two variants of G.723.1. They both operate on audio frames of 30 milliseconds (i.e. 240 samples), but the algorithms differ. The bitrate of the first variant is 6.4 kbit/s and the MOS is 3.9. The bitrate of the second variant is 5.3 kbit/s with MOS=3.7. The encoded frames for the two variants are 24 and 20 bytes long, respectively.

G.723.1 is a licensed codec, the last patent that covers it is expected to expire in 2014.

GSM 06.10
(also known as GSM Full Rate) is a codec designed by the European Telecommunications Standards Institute for use in the GSM mobile networks. This variant of the GSM codec can be freely used so you will often find it in open source VoIP applications. The codec operates on audio frames 20 milliseconds long (i.e. 160 samples) and it compresses each frame to 33 bytes, so the resulting bitrate is 13 kbit/s (to be precise, the encoded frame is exactly 32 and 1/2 byte, so 4 bits are unused in each frame). The codec's Mean Opinion Score is 3.7.

Speex
Speex is an open source patent-free codec designed by the Xiph.org Foundation. It is designed to work with sampling rates of 8 kHz, 16 kHz, and 32 kHz and can compress audio signal to bitrates between 2 and 44 kbit/s. For use in VoIP telephony, the most usual choice is the 8 kHz (narrow band) variant.

iLBC
iLBC (internet Low Bit Rate Codec) is a free codec developed by Global IP Solutions (later acquired by Google). The codec is defined in RFC3951. With iLBC, you can choose to use either 20 ms or 30 ms frames and the resulting bitrate is 15.2 kbit/s and 13.33 kbit/s, respectively. Much like Speex and GSM 06.10, you will find iLBC in many open source VoIP applications.


references:
http://toncar.cz/Tutorials/VoIP/VoIP_Basics_Overview_of_Audio_Codecs.html

MOS (Mean Opinion Score) value of a Codec

Previous article, we discussed about sampling of audio. Now having the audio signal represented as a sequence of samples, the next step is to compress it to reduce the consumption of network bandwidth required to transmit the speech to the receiving party.

The compression and decompression is handled by special algorithms we call codecs (COder-DECoder). Let's have a look at some popular codecs that are being used in Voice over IP. All the codecs we list here expect the input to be audio sampled at 8 kHz with 16-bit samples.

In the text below, we will mention the MOS of the codecs. MOS stands for "Mean Opinion Score". MOS measures the perceived quality of audio after it has been compresses by the particular codec, transmitted, and decompressed. The score is assigned by a group of listeners using the procedure specified in ITU-T standards P.800 and P.830. The interpretation of individual MOS values is as follows:

5 => Excellent
4 => Good
3 => Fair
2 => Poor
1 => Bad

G.711 MOS value = 4.2
G.729 MOS value is 4.0
G.723.1 variant 1 has MOS 3.9 and variant 2 has MOS 3.5
GSM 06.10 MOS is 3.7

references:
http://toncar.cz/Tutorials/VoIP/VoIP_Basics_Overview_of_Audio_Codecs.html 

VoIP Sampling and Quantization

When converting an analog signal (be it speech or another noise), you need to consider two important factors: sampling and quantization. Together, they determine the quality of the digitized sound.

Sampling is about the sampling rate — i.e. how many samples per second you use to encode the sound.

Quantization is about how many bits you use to represent each sample. The number of bits determines the number of different values you can represent with each sample.

Quantization

quantization is about how many bits you use to represent individual sound samples. In practice, we want to work with whole bytes, so let's consider 8 or 16 bits.

With 8-bit samples, each sample can represent 256 different values, so we can work with whole numbers between -128 and +127. Because of the whole numbers, it is inevitable that we introduce some noise into the signal as we convert it to digital samples. For example, if the exact analog value is "7.44125", we will represent it as "7". As we do this with each sample in the sequence, we slightly distort the signal — inject noise, in other words.

It turns out 8-bit samples do not result in a good quality. With only 256 sample values, the analog-to-digital conversion adds too much noise. The situation improves a lot if we switch to 16-bit samples as 16 bits give us 65536 different representations (from -32768 to +32767). 16-bit samples are what you will find on a CD and what VoIP codecs use as their input.


Sampling

With VoIP, you will most frequently encounter the sampling rate of 8 kilohertz. The frequency of 16 kHz can be used now and then in situations when a higher quality audio is required (with proportionally higher Internet bandwidth consumption).

The choice of sampling frequencies for the individual types of audio is not random. There is a rule (based on the work of Nyquist and Shanon) that the sampling frequency needs to be equal or greater than two times the transmitted bandwidth.

In short, It's good to remember that VoIP most frequently works with the sampling frequency of 8 kilohertz and each sample is stored in 16 bits.

references:
http://toncar.cz/Tutorials/VoIP/VoIP_Basics_Converting_Voice_to_Digital.html

WebRTC Architecture - A recap

Its been a long time looked into the WebRTC Stack. Never noticed earlier that NetEQ was part of the stack for Jitter buffer optimization. Here is the full stack.

WebRTC Native C++ API

An API layer that enables browser makers to easily implement the Web API proposal.

Transport / Session

The session components are built by re-using components from libjingle, without using or requiring the xmpp/jingle protocol.

RTP Stack

A network stack for RTP, the Real Time Protocol.

STUN/ICE

A component allowing calls to use the STUN and ICE mechanisms to establish connections across various types of networks.

Session Management

An abstracted session layer, allowing for call setup and management layer. This leaves the protocol implementation decision to the application developer.

VoiceEngine

VoiceEngine is a framework for the audio media chain, from sound card to the network.

iSAC / iLBC / Opus

iSAC: A wideband and super wideband audio codec for VoIP and streaming audio. iSAC uses 16 kHz or 32 kHz sampling frequency with an adaptive and variable bit rate of 12 to 52 kbps.

iLBC: A narrowband speech codec for VoIP and streaming audio. Uses 8 kHz sampling frequency with a bitrate of 15.2 kbps for 20ms frames and 13.33 kbps for 30ms frames. Defined by IETF RFCs 3951 and 3952.

Opus: Supports constant and variable bitrate encoding from 6 kbit/s to 510 kbit/s, frame sizes from 2.5 ms to 60 ms, and various sampling rates from 8 kHz (with 4 kHz bandwidth) to 48 kHz (with 20 kHz bandwidth, where the entire hearing range of the human auditory system can be reproduced). Defined by IETF RFC 6176. NetEQ for Voice

A dynamic jitter buffer and error concealment algorithm used for concealing the negative effects of network jitter and packet loss. Keeps latency as low as possible while maintaining the highest voice quality.

Acoustic Echo Canceler (AEC)

he Acoustic Echo Canceler is a software based signal processing component that removes, in real time, the acoustic echo resulting from the voice being played out coming into the active microphone.

Noise Reduction (NR)

The Noise Reduction component is a software based signal processing component that removes certain types of background noise usually associated with VoIP. (Hiss, fan noise, etc…)

VideoEngine

VideoEngine is a framework video media chain for video, from camera to the network, and from network to the screen.

VP8

Video codec from the WebM Project. Well suited for RTC as it is designed for low latency.

Video Jitter Buffer

Dynamic Jitter Buffer for video. Helps conceal the effects of jitter and packet loss on overall video quality.

Image enhancements

For example, removes video noise from the image capture by the webcam.

references:
https://webrtc.org/architecture/

What is WebRTC NetEQ

A dynamic jitter buffer and error concealment algorithm used for concealing the negative effects of network jitter and packet loss. Keeps latency as low as possible while maintaining the highest voice quality.

NetEQ is an implementation of a Jitter Buffer optimised for voice.

references:
https://webrtcglossary.com/neteq/