Tuesday, May 15, 2018

What is ISO-2 Country code?

This website gives information related to country codes used in various countries.
the iso-3166 is the specification which gives this. there is an extended profile to give the
details of sub regions as well. A snapshot of it is given below

It looks like the website is good one and the link is like below




references:
https://docs.oracle.com/cd/E13214_01/wli/docs92/xref/xqisocodes.html
https://www.iso.org/iso-3166-country-codes.html
https://www.iso.org/obp/ui/#search/code/

Friday, May 11, 2018

How does an RTCP packet structure look like?

RTCP (RTP Control protocol) is an application layer protocol for controlling the RTP. RTP based on UDP makes use of RTCP to make sure the packets are recontructed in order and to know the info about RTP packets.
Below is a block diagram for various field in an RTCP packet



Version: 2 bits:
The version of RTP. This value is same in RTP and RTCP packet

P, Padding bit
If set, contains some additional padding bytes at the end which are not part of control info. Last byte of padding is a count of how many padding bytes should be ignored. Padding is added for encryption algorithms with Fixed block sizes. In a compound RTCP packet, padding should be only required for the last individual packet because compound packet is encrypted as a whole.

Count:
This field is of 5 bits 0-31 . Contains the number of Reception reports contained  in the RTCP packet.

Type: 8 bits : this indicates the RTCP packet type. Various types of RTCP packets are FIR (Full Intra frame Request), NACK (negative acknowledgement),
SMPTEC,SMPTE time code mapping., IJ , extended inter Jitter report , SR sender report, RR receiver report, SDES source description, BYE good bye, APP application defined, RTPFB general RTP feedback, PSFB payload specific, XR RTCP extension, AVB AVB RTCP packet, RSI receiver summary info etc.

Length (16 bits). This is the length of RTCP packet   including the header and any padding bits.

references:
http://www.networksorcery.com/enp/protocol/rtcp.htm

Wednesday, May 9, 2018

Some very basics of VoIP - Part I

Happened to read the Cisco article and below some notes and some additions from other readings.

Below are the sections to highlight

1. Converting analog to digital form

Human speech frequency is anywhere between 200/300 Hx - 2700/2800Hz. The equipment supports maximum of 4Khz.

Below is a description of Nyquist theorem. According to this, the sampling frequency should be twice as the maximum frequency of analog signal.

Suppose the highest frequency component, in hertz, for a given analog signal is fmax. According to the Nyquist Theorem, the sampling rate must be at least 2fmax, or twice the highest analog frequency component. The sampling in an analog-to-digital converter is actuated by a pulse generator (clock). If the sampling rate is less than 2fmax, some of the highest frequency components in the analog input signal will not be correctly represented in the digitized output. When such a digital signal is converted back to analog form by a digital-to-analog converter, false frequency components appear that were not in the original analog signal. This undesirable condition is a form of distortion called aliasing.

Analysing a single byte in a sample which represents a sample.

e.g. 0011 1001

1. first bit from left (0) represents where the signal belongs, up or down of the X axis. in otherwods, -ve or +ve
2. 2-4th from left , i.e. 011 - represents the column window on y axis . for e.g between 1 - 2 or 2-3 etc
3. 5th to 8th bit represents the actual value between the window on y axis as per 2-4th bit

an 8K sampling require 8K * 8 (bits per byte) = 64 Kbit / sec. 64 Kbit/Sec is a good bandwidth. So most of the WAN apps are using the G.729 codec which require only 8 Kbps bandwidth. This is basically codec compression.

Some of the terms are:
MOS (Mean Opinion Score) : This is the score used to map the quality of voice produced at destination end as compared to source and its value ranges from 0 to 5.
For e.g. G.711 MOS is 4.1, G.729 MOS is 3.92. ILBC is 4.1

Some calculation is like this:
Codec Payload size per packet = {(codec bit rate/sec * sample size)/1000} bits
Sample size means length of sample or clipped sample length among the 8k samples from 1 sec analog wave i.e. 20 ms or 30 ms.
this makes the total packet size as below

Total packet size = Codec Payload + 12 Byte (RTP) + 8 Byte (UDP) + 20 Byte (IP) + 4 Byte (FR)
12 Byte (RTP) + 8 Byte (UDP) => Layer 4 header size
20 Byte (IP) => Layer 3 header size
4 Byte (FR) => Layer 2 Header size

Now how many packets are required to transmit 1 second long of data?

- in 1 second there are 1000 ms.
- We take samples every 20 ms and packetise it to trasmit over the network
- This means that to send 1000 ms data, we need 1000/20 = 50 packets

Now whats size of 1 packet?

We have one packet containing 20 ms of data by default.

We take samples of 20 ms and putting it in one packet.  in 20 ms, how many samples will be present? Every sample require 8 bit in binaries, i.e. a byte.
8000 samples / second means each 20 ms, we will have 8000/20 = 400 samples. each sample require 8 bits. Then it mean each 20 ms, we will have 400*8 = 3200 bits or 3.2 Kb.

RTP & RTCP : having a good comparison of bodyguard here. RTCP is a bodyguard which helps RTP packets to be re-arranged in a particular order.

Delay : Voice traffic is very sensible to delay. Cisco recommends maximum of 200ms of delay between source and destination. While ITU-T recommends it can be maximum of 150ms.

Compression: Two types of compression cRTP protocol, which is for compressing the headers. This is designed to reduce the IP/UDP/RTP headers to two bytes for most of the packets where no UDP checksums are being sent, or four bytes with Checksums. This follows RFC 2508 which is mainly depend on RFC 1144. cRTP specifies two formats :

- Compressed RTP (CR) => Used when IP, UDP, RTP headers remain consistent.
- Compressed UDP (CU) => used when there are large changes in the RTP timestamp or when the RTP payload type changes. The IP and UDP headers are compressed, RTP header is not.

PRI / BRI : PRI is typically used by Medium to large enterprises with Digital PBX telephone systems to provide digital access to PSTN. The B Channels may be used flexibly and re-assigned when necessary to meet special needs such as video conferences.


references:
https://learningnetwork.cisco.com/blogs/vip-perspectives/2014/11/20/first-date-with-voip
https://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-link-efficiency-mechanisms/22308-compression-qos.html
http://what-when-how.com/cisco-voice-over-ip-cvoice/voip-fundamentals-introducing-voice-over-ip-networks-part-2/

Monday, May 7, 2018

How Does the LAME Android SDK work

How do we encode and store a file in mp3 format? There are two ways to do

1. use a wav recorder and convert the byte buffer from the wav recorder into mp3 buffer on the fly
2. Read a wav file and convert it into mp3

#2 has a huge disadvantage of the original wav file being taking at least 100 MB for a 1 hr long file with just 8Kbps sampling.
#1 is efficient, but on the fly encoder needs  to be much efficient and faster.

The LAME mp3 encoder help one to achieve both. Below is how we do for on the fly encoding

int inSamplerate = 8000;
//get the minimum buffer value. Here we are considering only 8K samples
minBuffer = AudioRecord.getMinBufferSize(inSamplerate, AudioFormat.CHANNEL_IN_MONO,
                AudioFormat.ENCODING_PCM_16BIT);

//now create the AudioRecorder object.
  audioRecord = new AudioRecord(
                MediaRecorder.AudioSource.MIC, inSamplerate,
                AudioFormat.CHANNEL_IN_MONO,
                AudioFormat.ENCODING_PCM_16BIT, minBuffer * 2);


//now create a buffer to read in the bytes from recorder
short[] buffer = new short[inSamplerate * 2 * 5];

//now create the mp3 buffer.  'mp3buf' should be at least 7200 bytes long to hold all possible emitted data.
byte[] mp3buffer = new byte[(int) (7200 + buffer.length * 2 * 1.25)];

//create a file output stream for the path at which  the mp3 file should be written. 
 try {
            outputStream = new FileOutputStream(new File(filePath));
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }

//now create the AndroidLame object
androidLame = new LameBuilder()
                .setInSampleRate(inSamplerate)
                .setOutChannels(1)
                .setOutBitrate(32)
                .setOutSampleRate(inSamplerate)
                .build();

// now start the recording
 audioRecord.startRecording();

//now loop in and read the byte buffer.
while (isRecording) {
            bytesRead = audioRecord.read(buffer, 0, minBuffer);
            if (bytesRead > 0) {
                int bytesEncoded = androidLame.encode(buffer, buffer, bytesRead, mp3buffer);
                if (bytesEncoded > 0) {
                    try {
                        outputStream.write(mp3buffer, 0, bytesEncoded);
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                }
            }
        }


//now flush
int outputMp3buf = androidLame.flush(mp3buffer);

if (outputMp3buf > 0) {
         try {
                outputStream.write(mp3buffer, 0, outputMp3buf);
                outputStream.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
}

//now stop and release the recorder, and close the AndroidLame as well. 
 audioRecord.stop();
 audioRecord.release();
 androidLame.close();


references:
https://github.com/naman14/TAndroidLame 

How does the WAV format look like?

To write PCM samples captured from audio recorder to WAV file, below can be done

1. Create a Random access file.
2. write the wav header, which is as below

rFile.writeBytes("RIFF"); => This is chunk descriptor . 4 bytes
rFile.writeInt(0); => Chunk Size , 4 bytes => total size followed after this field including this => CHUNKSZ1
rFile.writeBytes("WAVE"); => 4 bytes
rFile.writeBytes("fmt "); => 4 bytes (note the space in here)
rFile.writeInt(Integer.reverseBytes(16)); => sub chunk size, 4 bytes
rFile.writeShort(Short.reverseBytes((short) 1)); => Audio format (PCM) => 2 bytes 
rFile.writeShort(Short.reverseBytes(channels)); => number of channels => 2 bytes
rFile.writeInt(Integer.reverseBytes(rate)); => Sampling rate => 4 bytes
rFile.writeInt(Integer.reverseBytes(rate * (resolution / 8) * mChannels)); =>  total bit rate (4 bytes)
rFile.writeShort(Short.reverseBytes((short) (channels * resolution / 8))); => block align (4 bytes)
rFile.writeShort(Short.reverseBytes(resolution)); => bits per sample (16 bit or 8 bit) , 2 byte
rFile.writeBytes("data"); => data chunk CHUNKSZ2
rFile.writeInt(0); => this is sub chunk size (the size of the remaining payload, which is also same as the total size followed by this)

Now if we have the PCM data, in order to put that in WAV format, write the PCM data appended to the random access file and then edit the CHUNKSZ1, CHUNKSZ2 values to the the total size of bytes. for e.g. like this below

// Write size to CHUNKSZ1
rFile.seek(4); // Write size to RIFF header
rFile.writeInt(Integer.reverseBytes(36 + payloadSz)); => 36 being the size of the header excluding the RIFF

rFile.seek(40); // Write size to CHUNKSZ2
rFile.writeInt(Integer.reverseBytes(payloadSz));

3. Close the Random access file.

Now we could make some funny things like repeat the whatever told multiple times in that case below can be done. Copy the contents from 44 bytes and then append to the random access file, come back and edit the payload size values again.

 RandomAccessFile wavFile = new RandomAccessFile(wavFilePath,"rw");
 long totalLength = wavFile.length();
 FileLogger.logInfo("Size of the wav file is "+totalLength);
 wavFile.seek(40);//get past the WAV headers
 //read the current size
 int subChunkSize = Integer.reverseBytes(wavFile.readInt());
 FileLogger.logInfo("subChunkSize is :"+subChunkSize);
 //now read the sub chunk size
 //buffer size
 int BUFF_SIZE = 1024;
 byte[] buffer = new byte[BUFF_SIZE];
 int readOffset = 44; // wav header end
 int countOfBytes  = -1;
 int debugCountOfByes = 0; // to analyze how many bytes were copied
 ByteArrayOutputStream bout = new ByteArrayOutputStream();
 wavFile.seek(readOffset);
 while ((countOfBytes = wavFile.read(buffer,0,BUFF_SIZE)) != -1 ){
  bout.write(buffer);
  readOffset += countOfBytes;
        debugCountOfByes += countOfBytes;
}
FileLogger.logInfo("Copied "+debugCountOfByes+" from file");
byte[] bytes = bout.toByteArray();
//now write this multiple times
for (int i = 0; i < count; i++){
     wavFile.write(bytes,44,bytes.length - 44);
}
int totalNewSize = subChunkSize + (count * bytes.length);
//now alter the wav header for the size values
wavFile.seek(4); // Write size to RIFF header
wavFile.writeInt(Integer.reverseBytes(36 + totalNewSize));

wavFile.seek(40); // Write size to Subchunk2Size field
wavFile.writeInt(Integer.reverseBytes(totalNewSize));
wavFile.close();

references:
http://soundfile.sapp.org/doc/WaveFormat/

Wednesday, May 2, 2018

What is LAME?

Lame is an MPEG Audio Layer 3 encoder (mp3) licensed under GPL.
Kudos to  Mike Cheng who started this development back in 1998.  However, in its current form Mark Taylor is the leader for LAME.
Today LAME is considered as best mp3 encoder at mid-high bit rates and at VBR. There are many projects that use LAME including famous WinAMP, many of the native operating systems.
Source is available for download at http://lame.sourceforge.net/download.php
LAME is only distributed in source code form. Many of the OS are supporting LAME compilation, which includes Windows, DOS, GNU/Linux, MacOS X, *BSD, Solaris, HP-UX, Tru64, Unix, AIX, Irix, NeXTStep, and so on..

to build the source code, one can follow the below

- Download the source and extract and navigate to the folder.
- Execute the below steps

$ ./configure
$ make
$ sudo make install


as part of the ./configure script execution, it creates make file for all of them mainly to notice for libmp3lame for i386, vector, dll, ACM, MacOS X, VS etc.

config.status: creating Makefile
config.status: creating libmp3lame/Makefile
config.status: creating libmp3lame/i386/Makefile
config.status: creating libmp3lame/vector/Makefile
config.status: creating frontend/Makefile
config.status: creating mpglib/Makefile
config.status: creating doc/Makefile
config.status: creating doc/html/Makefile
config.status: creating doc/man/Makefile
config.status: creating include/Makefile
config.status: creating Dll/Makefile
config.status: creating misc/Makefile
config.status: creating dshow/Makefile
config.status: creating ACM/Makefile
config.status: creating ACM/ADbg/Makefile
config.status: creating ACM/ddk/Makefile
config.status: creating ACM/tinyxml/Makefile
config.status: creating lame.spec
config.status: creating mac/Makefile
config.status: creating macosx/Makefile
config.status: creating macosx/English.lproj/Makefile
config.status: creating macosx/LAME.xcodeproj/Makefile
config.status: creating vc_solution/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
The source in zip form is only 1.3 Mb.

But make command resulted in below error

single_module -Wl,-exported_symbols_list,.libs/libmp3lame-symbols.expsym
Undefined symbols for architecture x86_64:
  "_lame_init_old", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libmp3lame.la] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Tried to compile using Xcode, but that was giving config.h is not found.
Gave the correct path reference to config.h and the error becomes similar to the terminal compilation, below

Undefined symbols for architecture x86_64:
  "_init_xrpow_core_sse", referenced from:
      _init_xrpow_core_init in quantize.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

so, the issue is in linking phase. The compilation is all good.

Looked at the ld command options, which displayed like this below. So all of the architectures are supported.

 ld -v
@(#)PROGRAM:ld  PROJECT:ld64-305
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
LTO support using: LLVM version 9.0.0, (clang-900.0.39.2) (static support for 21, runtime is 21)
TAPI support using: Apple TAPI version 900.0.15 (tapi-900.0.1

Now tried to comment the below lines and apparently that makes a successful build.

//#if defined(HAVE_XMMINTRIN_H)
//    if (gfc->CPU_features.SSE)
//        gfc->init_xrpow_core = init_xrpow_core_sse;
//#endif
//#ifndef HAVE_NASM
//#ifdef MIN_ARCH_SSE
//    gfc->init_xrpow_core = init_xrpow_core_sse;
//#endif
//#endif

So, the problem is really init_xrpow_core_sse. Then I stumbled upon the last link in this and this guy has also mentioned the same and mentions it to be due to the --host parameter.

Darwin myssytemcreds 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64

from the makefile, it looked like below

host = x86_64-apple-darwin17.2.0
host_alias =
host_cpu = x86_64
host_os = darwin17.2.0
host_vendor = apple



references:
http://lame.sourceforge.net/index.php
https://gist.github.com/trevorsheridan/1948448
https://stackoverflow.com/questions/21255976/how-to-solve-undefined-symbol-init-xrpow-core-sse-when-linking-lame-mp3-encode

Tuesday, May 1, 2018

What is PM2 ? - Some quick learning notes

From the definition from the official site, it appeared like below.

PM2 empowers your process management workflow. It allows you to fine-tune the behavior, options, environment variables, logs files of each application via a process file. It’s particularly useful for micro-service based applications.

Configuration format supported are Javascript, JSON and YAML.

what mattered to me most as part of learning is

- Ability to keep the code running even if it crashes out due to some reason
- Ability to get the log files

Basic usage is :
pm2 start app.js
To install, follow the below
npm install pm2@latest -g

It seems we can create a configuration file to manage multiple applications. Like below

process.yml

apps:
  - script   : app.js
    instances: 4
    exec_mode: cluster
  - script : worker.js
    watch  : true
    env    :
      NODE_ENV: development
    env_production:
      NODE_ENV: production


pm2 start process.yml
Next much useful thing about the PM2 was it as Process Management tool.

PM2 manages application states so that it can start, stop, restart and delete processes.
Some useful commands are :

pm2 start app.js --name "my-api"
pm2 start web.js --name "web-interface"
pm2 restart web-interface
pm2 stop web-interface

to list all running processes pm2 list
pm2 show 0
pm2 list --sort name:desc
pm2 list --sort [name|id|pid|memory|cpu|status|uptime][:asc|desc]

PM2 allows to restart an application based on a memory limit.
pm2 start big-array.js --max-memory-restart 20M

Other interesting thing about the PM2 was the folder structure especially to redirect the application logs to the file. Below are the directory structure

$HOME/.pm2 will contain all PM2 related files
$HOME/.pm2/logs will contain all applications logs
$HOME/.pm2/pids will contain all applications pids
$HOME/.pm2/pm2.log PM2 logs
$HOME/.pm2/pm2.pid PM2 pid
$HOME/.pm2/rpc.sock Socket file for remote commands
$HOME/.pm2/pub.sock Socket file for publishable events
$HOME/.pm2/conf.js PM2 Configuration


references:
http://pm2.keymetrics.io/docs/usage/quick-start/
http://pm2.keymetrics.io/docs/usage/process-management/#max-memory-restart