Why is a GPU preferable over a CPU for Machine Learning?

When buying a GPU for machine learning, there are several factors to consider. Here are some key aspects to look into:

GPU Architecture: The architecture of the GPU is crucial as it determines its computational capabilities and performance for machine learning tasks. Look for modern architectures, such as NVIDIA's Turing or Ampere, which offer dedicated hardware for machine learning workloads.

CUDA Cores: CUDA cores are parallel processors within the GPU that perform the heavy lifting for machine learning computations. More CUDA cores generally lead to faster training and inference times. Consider GPUs with a higher number of CUDA cores for improved performance.

Memory (VRAM): The amount of video RAM (VRAM) on the GPU is critical for deep learning models, especially those with larger datasets or complex architectures. Choose a GPU with sufficient VRAM to accommodate your training data and model requirements. Aim for at least 8GB or more of VRAM for most machine learning tasks.

Memory Bandwidth: The memory bandwidth of the GPU affects how quickly data can be read from and written to the VRAM. Higher memory bandwidth allows for faster data transfers, which can improve overall training performance.

Tensor Cores (for AI-specific workloads): Tensor cores are specialized hardware components found in some GPUs, such as NVIDIA's RTX series. They accelerate matrix operations commonly used in deep learning, offering significant performance gains. If you'll be working with AI-specific workloads, consider GPUs with tensor cores.

Compatibility and Software Support: Ensure that the GPU you choose is compatible with the deep learning frameworks and libraries you plan to use, such as TensorFlow or PyTorch. Also, check for reliable driver support and compatibility with your operating system.

Power and Cooling: Consider the power requirements of the GPU and ensure that your system's power supply can handle it. Additionally, check if your system has adequate cooling to handle the GPU's thermal requirements, as machine learning workloads can generate substantial heat.

Budget: Finally, consider your budget and strike a balance between performance and cost. Higher-end GPUs tend to offer better performance but come at a higher price. Evaluate your specific needs and choose a GPU that meets your requirements without exceeding your budget.

It's worth noting that GPU selection depends on the specific machine learning tasks you'll be performing. For more complex models or larger datasets, a higher-end GPU with more resources is generally recommended. However, for simpler models or smaller datasets, a mid-range GPU may suffice.

A CPU (Central Processing Unit) is the workhorse of your computer, and importantly is very flexible. It can deal with instructions from a wide range of programs and hardware, and it can process them very quickly. To excel in this multitasking environment a CPU has a small number of flexible and fast processing units (also called cores).

A GPU (Graphics Processing Unit) is a little bit more specialised, and not as flexible when it comes to multitasking. It is designed to perform lots of complex mathematical calculations in parallel, which increases throughput. This is achieved by having a higher number of simpler cores, sometimes thousands, so that many calculations can be processed all at once.

This requirement of multiple calculations being carried out in parallel is a perfect fit for:

graphics rendering — moving graphical objects need their trajectories calculated constantly, and this requires a large amount of constant repeat parallel mathematical calculations.

machine and deep learning — large amounts of matrix/tensor calculations, which with a GPU can be processed in parallel.

any type of mathematical calculation that can be split to run in parallel.

Tensor Processing Unit (TPU)

With the boom in AI and machine/deep learning there are now even more specialised processing cores called Tensor cores. These are faster and more efficient when performing tensor/matrix calculations. Exactly what you need for the type of mathematics involved in machine/deep learning.

Although there are dedicated TPUs, some of the latest GPUs also include a number of Tensor cores, as you will see later in this article.

Nvidia vs AMD

Nvidia’s GPUs have much higher compatibility, and are just generally better integrated into tools like TensorFlow and PyTorch.

trying to use an AMD GPU with TensorFlow requires using additional tools (ROCm), which tend to be a bit fiddly, and sometimes leave you with a not quite up to date version of TensorFlow/PyTorch, just so you can get the card working.

CUDA Cores and Tensor Cores

This is fairly simple really. The more CUDA (Compute Unified Device Architecture) cores / Tensor cores the better.

RAM and chip architecture should probably be considered first, and then look at cards with the highest number of CUDA/tensor cores from your narrowed down selection

For machine/deep learning Tensor cores are better (faster and more efficient) than CUDA cores. This is due to them being designed precisely for the calculations that are required in the machine/deep learning domain.

The reality is it doesn’t matter a great deal, CUDA cores are plenty fast enough. If you can get a card which includes tensor cores too, that is a good plus point to have, just don’t get too hung up on it.

CUDA cores — these are the physical processors on the graphics cards, typically in their thousands.

CUDA 11 — The number may change, but this is referring to the software/drivers that are installed to allow the graphics card to work. New releases are made regularly, and it can be installed like any other software.

CUDA generation (or compute capability) — this describes the capability of the graphics card in terms of it’s generational features. This is fixed in hardware, and so can only be changed by upgrading to a new card. It is distinguished by numbers and a code name. Examples: 3.x [Kepler], 5.x [Maxwell], 6.x [Pascal], 7.x [Turing] and 8.x [Ampere].

references:

https://towardsdatascience.com/how-to-pick-the-best-graphics-card-for-machine-learning-32ce9679e23b

What is ELI5 a long-form question answering dataset?

ELI5 is a dataset for long-form question answering. It contains 270K complex, diverse questions that require explanatory multi-sentence answers. Web search results are used as evidence documents to answer each question. ELI5 is also a task in Dodecadialogue.

The dataset comprises 270K threads from the Reddit forum ``Explain Like I'm Five'' (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline. However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

References:

https://paperswithcode.com/paper/eli5-long-form-question-answering

What is Streamlit app

Streamlit is an open-source Python library used for building interactive web applications and data visualizations. It simplifies the process of creating web-based interfaces for data analysis and machine learning tasks.

With Streamlit, you can write Python scripts that allow you to create custom web applications quickly and easily. You can incorporate charts, tables, interactive widgets, and other visualizations to present and explore your data.

Here's a simple example of a Streamlit application that displays a plot:

import streamlit as st

import pandas as pd

import matplotlib.pyplot as plt

# Load data

data = pd.read_csv("data.csv")

# Display plot

st.line_chart(data)

# Run the Streamlit app

if __name__ == "__main__":

    st.title("My Streamlit App")

To accept text input using Streamlit, you can use the text_input function provided by the Streamlit library. Here's an example of how to accept text input from the user:

import streamlit as st

# Accept text input

user_input = st.text_input("Enter your name", "John Doe")

# Display the input

st.write("Hello,", user_input)

After installing Xcode 14.3 in order to run my app on my iOS 16.3 iPhone XS. I get the following error:

Add the below code to the Podfile. It works for me. Version 14.3 beta 2 (14E5207e)

post_install do |installer|

    installer.generated_projects.each do |project|

          project.targets.each do |target|

              target.build_configurations.each do |config|

                  config.build_settings['IPHONEOS_DEPLOYMENT_TARGET'] = '13.0'

               end

          end

   end

end

Also, had to do remove all the pods and then reinstall the pods to make it work 

references:

https://stackoverflow.com/questions/75574268/missing-file-libarclite-iphoneos-a-xcode-14-3

What is Mel-Spectrogram

A signal is a variation in a certain quantity over time. For audio, the quantity that varies is air pressure. How do we capture this information digitally? We can take samples of the air pressure over time. The rate at which we sample the data can vary, but is most commonly 44.1kHz, or 44,100 samples per second. What we have captured is a waveform for the signal, and this can be interpreted, modified, and analyzed with computer software.

import librosa

import librosa.display

import matplotlib.pyplot as plt

y, sr = librosa.load('./example_data/blues.00000.wav')

plt.plot(y);

plt.title('Signal');

plt.xlabel('Time (samples)');

plt.ylabel('Amplitude');

This is great! We have a digital representation of an audio signal that we can work with. Welcome to the field of signal processing! You may be wondering though, how do we extract useful information from this? It looks like a jumbled mess. This is where our friend Fourier comes in.

The Fourier Transform

An audio signal is comprised of several single-frequency sound waves. When taking samples of the signal over time, we only capture the resulting amplitudes. The Fourier transform is a mathematical formula that allows us to decompose a signal into it’s individual frequencies and the frequency’s amplitude. In other words, it converts the signal from the time domain into the frequency domain. The result is called a spectrum.

This is possible because every signal can be decomposed into a set of sine and cosine waves that add up to the original signal. This is a remarkable theorem known as Fourier’s theorem. Click here if you want a good intuition for why this theorems is true. There is also a phenomenal video by 3Blue1Brown on the Fourier Transform if you would like to learn more here.

The fast Fourier transform (FFT) is an algorithm that can efficiently compute the Fourier transform. It is widely used in signal processing. I will use this algorithm on a windowed segment of our example audio.

import numpy as np

n_fft = 2048

ft = np.abs(librosa.stft(y[:n_fft], hop_length = n_fft+1))

plt.plot(ft);

plt.title('Spectrum');

plt.xlabel('Frequency Bin');

plt.ylabel('Amplitude');

The Spectrogram

The fast Fourier transform is a powerful tool that allows us to analyze the frequency content of a signal, but what if our signal’s frequency content varies over time? Such is the case with most audio signals such as music and speech. These signals are known as non periodic signals. We need a way to represent the spectrum of these signals as they vary over time. You may be thinking, “hey, can’t we compute several spectrums by performing FFT on several windowed segments of the signal?” Yes! This is exactly what is done, and it is called the short-time Fourier transform. The FFT is computed on overlapping windowed segments of the signal, and we get what is called the spectrogram. Wow! That’s a lot to take in. There’s a lot going on here. A good visual is in order.

You can think of a spectrogram as a bunch of FFTs stacked on top of each other. It is a way to visually represent a signal’s loudness, or amplitude, as it varies over time at different frequencies. There are some additional details going on behind the scenes when computing the spectrogram. The y-axis is converted to a log scale, and the color dimension is converted to decibels (you can think of this as the log scale of the amplitude). This is because humans can only perceive a very small and concentrated range of frequencies and amplitudes.

spec = np.abs(librosa.stft(y, hop_length=512))

spec = librosa.amplitude_to_db(spec, ref=np.max)

librosa.display.specshow(spec, sr=sr, x_axis='time', y_axis='log');

plt.colorbar(format='%+2.0f dB');

plt.title('Spectrogram');

The Mel Scale

Studies have shown that humans do not perceive frequencies on a linear scale. We are better at detecting differences in lower frequencies than higher frequencies. For example, we can easily tell the difference between 500 and 1000 Hz, but we will hardly be able to tell a difference between 10,000 and 10,500 Hz, even though the distance between the two pairs are the same.

In 1937, Stevens, Volkmann, and Newmann proposed a unit of pitch such that equal distances in pitch sounded equally distant to the listener. This is called the mel scale. We perform a mathematical operation on frequencies to convert them to the mel scale.

References

https://medium.com/analytics-vidhya/understanding-the-mel-spectrogram-fca2afa2ce53

What is OpenAI Whisper

Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. We show that the use of such a large and diverse dataset leads to improved robustness to accents, background noise and technical language. Moreover, it enables transcription in multiple languages, as well as translation from those languages into English. We are open-sourcing models and inference code to serve as a foundation for building useful applications and for further research on robust speech processing.

The Whisper architecture is a simple end-to-end approach, implemented as an encoder-decoder Transformer. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.

Other existing approaches frequently use smaller, more closely paired audio-text training datasets,1 2,3 or use broad but unsupervised audio pretraining.4,5,6 Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper’s zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.

references:

https://openai.com/research/whisper

What is TikToken Library

Tiktoken is an open-source tool developed by OpenAI that is utilized for tokenizing text.

Tokenization is when you split a text string to a list of tokens. Tokens can be letters, words or grouping of words (depending on the text language).

For example, “I’m playing with AI models” can be transformed to this list [“I”,”’m”,” playing”,” with”,” AI”,” models”].

Then these tokens can be encoded in integers.

OpenAI uses a technique called byte pair encoding (BPE) for tokenization. BPE is a data compression algorithm that replaces the most frequent pairs of bytes in a text with a single byte. This reduces the size of the text and makes it easier to process.

You can use tiktoken to count tokens, because:

You need to know whether the text your are using is very long to be processed by the model

You need to have an idea about OpenAI API call costs (The price is applied by token).

For example, if you are using GPT-3.5-turbo model you will be charged: $0.002 / 1K tokens

How to count the number of tokens using tiktoken?

pip install tiktoken

import tiktoken

Encoding

Different encodings are used in openai: cl100k_base, p50k_base, gpt2.

These encodings depend on the model you are using:

For gpt-4, gpt-3.5-turbo, text-embedding-ada-002, you need to use cl100k_base.

All this information is already included in OpenAI API, you don’t need to remember it. Therefore, you can call the encoding using 2 methods:

If you know the exact encoding name:

encoding = tiktoken.get_encoding("cl100k_base")

Alternatively, you can allow the OpenAI API to provide a suitable tokenization method based on the model you are using:

encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")

print(encoding)

Tokenization

Let’s tokenize this text:

text = "I'm playing with AI models"

This will return a list of tokens integer:

tokens_integer=encoding.encode(text)

tokens_integer

[40, 2846, 5737, 449, 15592, 4211]

print(f"{len(tokens_integer)} is the number of tokens in my text")

6 is the number of tokens in my text

It’s worth mentioning that we can obtain the corresponding token string for each integer token by utilizing the ‘encoding.decode_single_token_bytes()’ function (each string will be a bytes ‘b’ string)

tokens_string = [encoding.decode_single_token_bytes(token) for token in tokens_integer]

tokens_string

[b'I', b"'m", b' playing', b' with', b' AI', b' models']

the space before each word? This is how it works in OpenAI with tiktoken.

Count the number of token in the message to be sent using the API:

message =[{

   "role": "user",

   "content": "Explain to me how tolenization is working in OpenAi models?",

   }]

tokens_per_message = 4 

# every message follows <|start|>{role/name}\n{content}<|end|>\n

num_tokens = 0

num_tokens += tokens_per_message

for key, value in message[0].items():

   text=value

   num_tokens+=len(encoding.encode(value))

   print(f"{len(encoding.encode(value))} is the number of token included in {key}")

num_tokens += 3

# every reply is primed with <|start|>assistant<|message|>

print(f"{num_tokens} number of tokens to be sent in our request")

1 is the number of token included in role

15 is the number of token included in content

23 number of tokens to be sent in our request

import openai

openai.api_key='YOUR_API_KEY'

response = openai.ChatCompletion.create(

     model='gpt-3.5-turbo-0301',

     messages=message,

     temperature=0,

     max_tokens=200 

 )

num_tokens_api = response["usage"]["prompt_tokens"]

print(f"{num_tokens_api} number of tokens used by the API")

23 number of tokens used by the API

The number of tokens is the same as what we calculated using ‘tiktoken’.

Furthermore, let’s count the number of tokens in ChatGPT answer :

resp=response["choices"][0]["message"].content

len(encoding.encode(resp))

200