What are differences between Qwen and QuantFactory LLM Models

Model Source:

Qwen: Developed by the Qwen project, a large language model research effort focused on transformer-based models.

QuantFactory GGUF: Created by QuantFactory, a company specializing in optimizing and deploying large language models.

Focus:

Qwen: Primarily focuses on the underlying model architecture and training process, aiming to achieve high performance and capabilities.

QuantFactory GGUF: Leverages the Qwen model as a base and emphasizes optimizing it for deployment through quantization and conversion to the GGUF format.

Quantization:

Qwen: Might offer base, unquantized models for research purposes.

QuantFactory GGUF: Specifically focuses on providing quantized versions of Qwen models in the GGUF format. Quantization reduces model size and memory footprint, making it more efficient to run on resource-constrained hardware, like local machines with GPUs.

Target Users:

Qwen: Primarily targets researchers and developers interested in exploring and customizing the model architecture and functionalities.

QuantFactory GGUF: Caters to users who need a deployable version of the Qwen model for practical applications on resource-limited hardware.

Availability:

Qwen: Models might be available through the Hugging Face model hub or the Qwen project website (depending on the specific model version).

QuantFactory GGUF: Models might be available on the Hugging Face model hub or through QuantFactory's resources (specific distribution details depend on the company's policies).

References:

Gemini 

Compatibility & supported file formats:

 Llama.cpp (by Georgi Gerganov)

GGUF (new)

GGML (old)

Transformers (by Huggingface)

bin (unquantized)

safetensors (safer unquantized)

safetensors (quantized using GPTQ algorithm via AutoGPTQ)

AutoGPTQ (quantization library based on GPTQ algorithm, also available via Transformers)

safetensors (quantized using GPTQ algorithm)

koboldcpp (fork of Llama.cpp)

bin (using GGML algorithm)

ExLlama v2 (extremely optimized GPTQ backend for LLaMA models)

safetensors (quantized using GPTQ algorithm)

AWQ (low-bit quantization (INT3/4))

safetensors (using AWQ algorithm)

Notes:

* GGUF contains all the metadata it needs in the model file (no need for other files like tokenizer_config.json) except the prompt template

* llama.cpp has a script to convert *.safetensors model files into *.gguf

* Transformers & Llama.cpp support both CPU, GPU and MPU inference

What are the differences between various Llamaa3 models

The Llamaa 3 models being compared are the ones below: 

meta-llama/Meta-Llama-3-8B

Meta-Llama-3-8B-Instruct

Meta-Llama-3-70B-Instruct

Meta-Llama-3-70B

The main differences between the Meta Llama-3 models you listed lie in their size and fine-tuning:

Meta-Llama-3-8B and Meta-Llama-3-70B: These terms refer to the size of the models, measured in billions of parameters. 8B signifies 8 billion parameters, while 70B signifies 70 billion parameters. Generally, larger models have a higher capacity for complex tasks and potentially better performance on benchmarks. However, they also require more powerful hardware and computational resources to run.

Fine-Tuning:

Base Models (Meta-Llama-3-8B and Meta-Llama-3-70B): These are the foundational models pre-trained on a massive dataset of text and code. They are versatile and can be used for various tasks like text generation, translation, and question answering.

Instruct Fine-Tuned Models (Meta-Llama-3-8B-Instruct and Meta-Llama-3-70B-Instruct): These models are built upon the base models but have received additional training specifically focused on datasets containing instructions and human demonstrations. This fine-tuning enhances their ability to follow instructions and complete tasks as directed. They are potentially better suited for tasks like writing different kinds of creative content or following specific coding prompts.

The best model for you depends on your specific needs:

Task Complexity: For simpler tasks like summarization, the 8B model might suffice. Complex tasks like advanced code generation might benefit from the 70B model.

Computational Resources: If you have limited resources, the 8B model requires less power to run.

Performance vs. Speed: The 8B model might be faster, but the 70B model could offer better performance if speed isn't a major concern.

Need for Following Instructions: If your task heavily relies on following instructions, the Instruct fine-tuned models (8B or 70B) would be a better choice.

Remember, Meta might offer additional resources or documentation comparing these models in more detail. It's always recommended to consult those resources for the latest information.

What is Jurassic Jumbo model

What is Jurassic Jumbo model 

Jurassic-1 Jumbo is a 178B parameter auto-regressive language model developed by AI21 Labs. It is the largest and most sophisticated language model ever released for general use by developers. Jurassic-1 Jumbo can perform a wide range of tasks, including:

Generating text, translating languages, writing different kinds of creative content, and answering your questions in an informative way.

Summarizing or simplifying text.

Writing different kinds of creative content, such as poems, code, scripts, musical pieces, email, letters, etc.

Answering questions in a comprehensive and informative way, even if they are open ended, challenging, or strange.

Jurassic-1 Jumbo Architecture

Jurassic-1 Jumbo is based on the Transformer architecture, which is a state-of-the-art neural network architecture for natural language processing. The Transformer architecture is composed of self-attention modules, which allow the model to learn long-range dependencies in text.

Jurassic-1 Jumbo also uses a number of other techniques to improve its performance, including:

A large vocabulary: Jurassic-1 Jumbo has a vocabulary of over 100 billion tokens, which allows it to represent a wide range of human language.

A deep architecture: Jurassic-1 Jumbo has 76 layers, which allows it to learn complex relationships in text.

A large training dataset: Jurassic-1 Jumbo was trained on a massive dataset of text and code, which allows it to perform a wide range of tasks.

References:

https://groups.google.com/g/react-js-for-front-end-development/c/_ABOBwavIP4?pli=1

Setting up Llama-3 locally using OLlama

To setup Llama-3 locally, we will use Ollama — an open-source framework that enables open-source Large Language Models (LLMs) to run locally in computer.

CPU: Any modern CPU with at least 4 cores recommended for running smaller models. For running 13B models, CPU with at least 8 cores is recommended. GPU is optional for Ollama, but if available can improve the performance drastically.

RAM: At least 8 GB of RAM to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.

Disk Capacity: Recommend at least 12 GB of disk space available, to install Ollama and the base models. Additional space will be required if more models are planned to be installed.

Downloaded the Llama installation file from this link https://ollama.com/download

 It downloaded the Ollama dmg file. Installed it and ran the below command which downloaded the model file 

ollama run llama3

This is an 8B instruct model of Llama-3 

To download specific model,  llama3:70b  can be used 

Lamma-3 8B Instruct model, takes about ~4.7 GB download size.

On Mac, the model file is stored under ~/.ollama/models 

To upgrade ollama, below command can be used

ollama pull llama3

To remove below can be used 

ollama rm llama3

There are multiple prompting options. 

command-line: This is the simplest of all option. As we saw in Step-2, with the run command, Ollama command-line is ready to accept prompt messages. We can type in the prompt message there, to get Llama-3 responses, as shown below. To exit the conversation, type the command /bye.

ReST API (HTTP Request): As we saw in Step-1, Ollama is ready to serve Inference API requests, on local HTTP port 11434 (default). You can hit the Inference API endpoint with HTTP POST request containing the prompt message payload. Here is an example of a CURL request for a prompt

 curl -X POST http://localhost:11434/api/generate -d "{\"model\": \"llama3\",  \"prompt\":\"Tell me a good joke?\", \"stream\": false}"

{"model":"llama3","created_at":"2024-06-27T02:26:06.929468Z","response":"Here's one:\n\nWhy couldn't the bicycle stand up by itself?\n\n(wait for it...)\n\nBecause it was two-tired!\n\nHope that made you smile! Do you want to hear another one?","done":true,"done_reason":"stop","context":[128006,882,128007,271,41551,757,264,1695,22380,30,128009,128006,78191,128007,271,8586,596,832,1473,10445,7846,956,279,36086,2559,709,555,5196,1980,65192,369,433,62927,18433,433,574,1403,2442,2757,2268,39115,430,1903,499,15648,0,3234,499,1390,311,6865,2500,832,30,128009],"total_duration":10178937333,"load_duration":8604434667,"prompt_eval_count":16,"prompt_eval_duration":159149000,"eval_count":40,"eval_duration":1408267000}%            

stream” flag as “false” in the CURL request, to get all responses at once. The default value for “stream” is true, in which case, you will receive multiple HTTP responses with a streaming result of tokens. For the last response of the streaming results, the “done” attribute will be returned as “true”.

To execute via Python script, below can be done 

pip install langchain-community

from langchain_community.llms import Ollama

llm = Ollama(model="llama3")

prompt = "Tell me a joke about llama"

result = llm.invoke(prompt)

print(result)

# 'Why did the llama go to the party?\n\nBecause it was a hair-raising experience!'

References:

https://medium.com/@renjuhere/llama-3-running-locally-in-just-2-steps-e7c63216abe7

Comparison Evaluators in Langchain

Comparison evaluators in LangChain help measure two different chains or LLM outputs. These evaluators are helpful for comparative analyses, such as A/B testing between two language models, or comparing different versions of the same model. They can also be useful for things like generating preference scores for ai-assisted reinforcement learning.

These evaluators inherit from the PairwiseStringEvaluator class, providing a comparison interface for two strings - typically, the outputs from two different prompts or models, or two versions of the same model. In essence, a comparison evaluator performs an evaluation on a pair of strings and returns a dictionary containing the evaluation score and other relevant details.

evaluate_string_pairs: Evaluate the output string pairs. This function should be overwritten when creating custom evaluators.

aevaluate_string_pairs: Asynchronously evaluate the output string pairs. This function should be overwritten for asynchronous evaluation.

requires_input: This property indicates whether this evaluator requires an input string.

requires_reference: This property specifies whether this evaluator requires a reference label.

Often you will want to compare predictions of an LLM, Chain, or Agent for a given input. The StringComparison evaluators facilitate this so you can answer questions like:

Which LLM or prompt produces a preferred output for a given question?

Which examples should I include for few-shot example selection?

Which output is better to include for fine-tuning?

Below is sample code for this

def pairwise_comparison():

    evaluator = load_evaluator("labeled_pairwise_string")

    result = evaluator.evaluate_string_pairs(

        prediction="there are three dogs",

        prediction_b="4",

        input="how many dogs are in the park?",

        reference="four",

    )

    print("Evaluation result is ",result)

Output is something like below 

Evaluation result is  {'reasoning': "Both Assistant A and Assistant B provided direct answers to the user's question. However, Assistant A's response is incorrect as it stated there are three dogs in the park, while the user's question indicated there are four. On the other hand, Assistant B correctly answered the user's question by stating there are four dogs in the park. Therefore, Assistant B's response is more accurate and relevant to the user's question. \n\nFinal Verdict: [[B]]", 'value': 'B', 'score': 0}

References:

https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/comparison/

Metrics of RAG

Faithfulness : This measures the factual consistency of the generated answer against the given context. It is calculated from answer and retrieved context. The answer is scaled to (0,1) range. Higher the better.

The generated answer is regarded as faithful if all the claims that are made in the answer can be inferred from the given context. To calculate this a set of claims from the generated answer is first identified. Then each one of these claims are cross checked with given context to determine if it can be inferred from given context or not. The faithfulness score is given by divided by

Answer relevancy

The evaluation metric, Answer Relevancy, focuses on assessing how pertinent the generated answer is to the given prompt. A lower score is assigned to answers that are incomplete or contain redundant information and higher scores indicate better relevancy. This metric is computed using the question, the context and the answer.

The Answer Relevancy is defined as the mean cosine similarity of the original question to a number of artifical questions, which where generated (reverse engineered) based on the answer:

Context recall

Context recall measures the extent to which the retrieved context aligns with the annotated answer, treated as the ground truth. It is computed based on the ground truth and the retrieved context, and the values range between 0 and 1, with higher values indicating better performance.

To estimate context recall from the ground truth answer, each sentence in the ground truth answer is analyzed to determine whether it can be attributed to the retrieved context or not. In an ideal scenario, all sentences in the ground truth answer should be attributable to the retrieved context.

Context precision

Context Precision is a metric that evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. Ideally all the relevant chunks must appear at the top ranks. This metric is computed using the question, ground_truth and the contexts, with values ranging between 0 and 1, where higher scores indicate better precision.

Context relevancy

This metric gauges the relevancy of the retrieved context, calculated based on both the question and contexts. The values fall within the range of (0, 1), with higher values indicating better relevancy.

Context entity recall

Context recall measures the extent to which the retrieved context aligns with the annotated answer, treated as the ground truth. It is computed based on the ground truth and the retrieved context, and the values range between 0 and 1, with higher values indicating better performance.

To estimate context recall from the ground truth answer, each sentence in the ground truth answer is analyzed to determine whether it can be attributed to the retrieved context or not. In an ideal scenario, all sentences in the ground truth answer should be attributable to the retrieved context.

references:

https://docs.ragas.io/en/stable/concepts/metrics/index.html#