I am a student and I want answers from my textbooks easily so that I can understand the subject better.

Goal

1. Understand Vector Database

2. Understand RAG

3. Understand how to productionize a GENAI app to a certain degree

Ingestion

1. Get all PDFs from Class 11 Biology PDFs

2. Create a Qdrant vector database

3. Write Python Script to upload the PDFs into the database

4. Do not use frameworks such as Langchain / LLamaIndex

5. Insert Metadata such as Chapter , Page Number when you are inserting data into the Database

Github for Qdrant Code

This directory collects helper scripts, configs, and notebooks for working with a Qdrant vector store, ingesting data, and prototyping RAG workflows.

Search

1. User asks a question

2. Use the question to search the Database

3. Use simple text search to get results

4. Use semantic search to get results

5. Use hybrid search to get results

6. Understand the difference between text search / semantic search / hybrid search

7. In the search results show the meta data associated with the result

8. Advanced [ Implement RRF ]

LLM

1. User asks a question

2. Understand RAG

3. Use the search results and the LLM to get the answer

4. In the answer , show the portions of the text used to frame the answer [ the Chapter , Page Number ]

5. Understand how effectively the LLM and RAG is answering the question

UI

1. Create screens to upload more documents

2. Create screens to have the user ask a question

3. Create a chatbot

4. Create a chatbot with memory

FastAPI

1. Use FASTAPI to expose the function

Docker

1. Make a docker image of the FASTAPI

Advanced

Langraph

1. Understand Langraph using the repo https://github.com/ambarishg/langchain_and_langraph

2. Inspiration from the LangGraph Complete Course for Beginners – Complex AI Agents with Python

Agent Framework

1. Understand the Agent Framework using the repo https://github.com/ambarishg/agent-framework

2. Inspiration from Microsoft Agent framework samples

Word embeddings give us a way to use an efficient, dense representation in which similar words have a similar encoding.

Importantly, you do not have to specify this encoding by hand. An embedding is a dense vector of floating point values (the length of the vector is a parameter you specify).

Instead of specifying the values for the embedding manually, they are trainable parameters (weights learned by the model during training, in the same way a model learns weights for a dense layer).

It is common to see word embeddings that are 8-dimensional (for small datasets), up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but takes more data to learn.

Let us explore the word embedding with some examples. We will use spacy for demonstration.

import numpy as np
import spacy
from sklearn.metrics.pairwise import cosine_similarity
# Need to load the large model to get the vectors
nlp = spacy.load('en_core_web_lg')

nlp("queen").vector.shape

We find the word embedding of a single word queen and find that we have a vector with 1 row and 300 columns. Therefore a single word is converted to 300 numerical values.

We find the similarity between the words using cosine similarity

cosine_similarity([nlp("queen").vector],[nlp("king").vector])

0.725261

cosine_similarity([nlp("queen").vector],[nlp("mother").vector])

0.44720313

cosine_similarity([nlp("queen").vector],[nlp("princess").vector])

0.6578181

We observe that the similarity between queen and king is the highest , followed by princess and mother

We will see that how we can use the similarity between sentences

x1 = nlp("I am a software consultant").vector
x2 = nlp("Hey ,me  data guy").vector
x3 = nlp("Hey ,me  plumber").vector

x1.shape , x2.shape , x3.shape

((300,), (300,), (300,))

We find that the shape of the sentence vectors are also 1 x 300. The individual words also have shape 1 x 300 . But for a sentence , we average the vectors so as to get the shape also as 1 x 300.

cosine_similarity([nlp("x1").vector],[nlp("x2").vector])

0.7383951

cosine_similarity([nlp("x1").vector],[nlp("x3").vector])

0.64217263

We see that the similarity between the sentence with software consultant and data guy is higher than the sentence with software consultant and plumber

User queries cannot be answered satisfactorily using a single source. We can use a very effective technique called RAG( Retrieval Augmented Generation) for getting the results from a rich context and sending it to a LLM( Large Language Model ). However , the results can be improved if the answer is enriched using other sources.The answer to this problem is combining the power of Agents which would help the RAG to combine other sources and provide an appropriate answer.

Let us take a concrete example. You are company which does Field Service Management. Your employees fix things. You have developed a mobile application which helps the field engineers to ask questions on a knowledge repository [ You have used a Vector Database to load all your documents ]

What happens if the information required is not present in the Vector database ?

Here is where Agentic RAG comes in

The Agent would first search in the Vector Database and then it would try to find answers from different sources from the enterprise such as [ Databases , Applications , File Stores ].

The Agent would use Tools to get this information.

Flow of the Blog

The Blog follows the Concept to Code philosophy

The first part is conceptual and is meant for all , executives and all technical audience. The second part goes into the implementation details targetting the technical audience.

Agentic RAG

Agentic RAG combines the power of RAG with AI Agents.

Simple RAG

Let us first look at the simple RAG

Steps

1. The user provides the Query

2. The query is used to search in the Vector Database

3. The results are returned to the orchestrator which sends the results to the LLM

4. The LLM utilizes the results to provide the answer

Agentic RAG

Steps

1. The user provides the Query

2. The query is used to search in the Vector Database

3. The results are returned to the orchestrator which sends the results to the LLM

4. The LLM utilizes the results to provide the answer.

4a. Answer sent by the LLM is satisfactory, you are OK and no need to go to the next steps

5. Answer sent by the LLM is not satisfactory

6. Send the user input to the Agent

7. The Agent provides the answer

We can optimize this data flow a bit by not sending the results to the LLM at first. We can check if the search results are satisfactory before sending it to the LLM. The modified steps are as follows

1. The user provides the Query

2. The query is used to search in the Vector Database

3. The results are returned to the orchestrator which sends the results to the LLM

4. Check the search results sent by the Vector Database is satisfactory or not

5. If the search results are satisfactory send the search results to the LLM

5a. The LLM utilizes the results to provide the answer

6. If the search results are NOT satisfactory follow the next steps

7. Send the user input to the Agent

8. The Agent provides the answer

Agent

An Agent is a composite system.

It has the following components

1. LLM

2. Tools

3. Planning

4. Memory

We would be using Crewai as the agentic framework for this blog.

Crewai

Crewai has the following components

• Crew

  • Top Level Organization

  • Manages Agents

  • Ensures collaboration

• Agents

  • Specialized Team members

  • Have specific Roles

  • Has Tasks associated with it

  • Has Tools

• Tasks

  • Uses specific tools

  • Has Individual assignments

Crewai has other components but for this discussion we would like to keep this simple and restrict ourselves to these components only

This agentic RAG would be done by Crewai using

1. Crew

2. 1 Simple Agent

3. 1 Task

4. 2 tools [ A tool to search the Vector Database , another tool to search the Internet ]

When the user enters the query , the agent takes control and is responsible for using the Tasks and the Tools

Agent

The Agent is a Router Assistant .

You are an experienced assistant specializing in routing the flow to the appropriate tool.

Tasks

The Agent uses the Task. The Task does this

Based on the user's questions: ask the VECTOR DB TOOL. If the answer is not found, ask the SEARCH TOOL for the answer

Let us revisit

Note that the decision of choosing the VECTOR DB Tool or the Search Tool is taken by the Agent

Implementation details

For this blog, we are using the following

• Vector Database - Qdrant Cloud

• LLM - Azure Open AI

• Agent Framework - Crew AI

Crew AI Implementation details

# Configure the LLM
llm = LLM(model=os.getenv("model"),)

Step 1 - We have first have to configure the LLM.

Step 2 Next step is to define the Agent

# Define the Agent
router_agent = Agent(
    role='Router Assistant',
    goal='Assistant to route to the appropriate tool',
    backstory="You are an experienced assistant specializing in routing the flow to the appropriate tool.",
    verbose=True,
    allow_delegation=False,
    llm=llm,
    tools=[search_serper, rag_qdrant]  
)

Note the tools which we are using in this space

1. VECTOR DB tool = rag_qdrant

2. SEARCH tool = search_serper

Step 3 Define the Task

# Define the Task
task = Task(
    description=""""
    Based on the user's questions: {questions}, 
    ask the tool rag_qdrant. If the answer is not found, 
    ask the tool search for the answer 
    """,
    expected_output="A properly well worded answer from the tool which can be used by the user.",
    agent=router_agent
)

Step 4 Define the Crew

# Create the Crew
crew = Crew(
    agents=[router_agent],
    tasks=[task],
    verbose=True,
)

Step 4 Run the Crew

# User input for travel preferences
user_input = {
    "questions": "What is naked trust ?"
}

# Execute the Crew
result = crew.kickoff(inputs=user_input)

Tools Implementation Details

In this section we go into the details of the tools , Search Tool and the Vector DB Tool

Search TOOL

Let us explore in depth the Search Tool

@tool('SerperDevTool')
def search_serper(search_query: str):
    """Search the web for information on a given topic"""
    tool = SerperDevTool(
    search_url=SEARCH_URL
    n_results=2,
)

The Search URL is "https://google.serper.dev/search"

Here we are using the SerperDevTool for searching Google

VECTOR DB TOOL

@tool('RAG_QDRANT')
def rag_qdrant(search_query:str):
    """Gets the answer from Qdrant"""
    rag_helper = RAGSystem(AzureOpenAIManager())
    return (rag_helper.query(search_query))

Vector DB Tool is the Tool which uses the Qdrant Vector DB and Azure Open AI as the LLM to search as well as utilizes the LLM to get the result. We will in the subsequent sections go deeper and see the internals of the RAGSystem class

RAG System Details

It has 2 components

• Retrieve - retrieve results from the Qdrant DB

• Query - get the response from the user query using the Vector DB and the LLM.

import os
import logging

class RAGSystem:
    def __init__(self, generator: ILLMHelper):
        self.generator = generator
        # Initialize Qdrant components
        logging.basicConfig(level=logging.INFO)
        logging.info("Loading model: %s", os.getenv("MODEL_NAME"))


    def retrieve(self, query: str) -> str:
        self.model = SentenceTransformer(os.getenv("MODEL_NAME"))
        self.client = qc.QdrantClient(
            url=os.getenv("QDRANT_URL"),
            api_key=os.getenv("QDRANT_KEY")
        )
        vector = self.model.encode(query, convert_to_tensor=True).tolist()
        results = self.client.search(
            collection_name=os.getenv("QDRANT_COLLECTION"),
            query_vector=vector,
            limit=5
        )
        return "\n---\n".join([r.payload["content"] for r in results])

    def query(self, query: str) -> str:
        context = self.retrieve(query)
        return self.generator.generate(context, query)

Init method

Initializes the LLM to be used by the class.

In this blog we have used the Azure Open AI model

Retrieve method

1. Uses SentenceTransformer model to create the embedding

2. Searches the VECTOR DB using the embedding

3. Gets the results from the embedding

Query method

1. Uses the retrieve method to get the context

2. Uses the context and the query to get the response

Github Repo for this Blog is also provided

Through the endless participation in these competitions , I had got the following treasures

• Discovered new ways of thinking. Thinking is perhaps the most important skill nowadays

• Discovered communities and all the nice people supporting the community organizing the hackathon. Having a feeling that you belong to a good #community is very surreal. My gratitude to all community builders / community leads for making our learning inclusive.

What is more important is what happens after the competition. I prefer participating in competitions where the winning solutions are shared [ what can be more beneficial , is the code is also shared]

This is what I plan or usually do. There is always a difference between plan and execution. At least having a plan helps in the execution

• Congratulate the winners and everyone. Express gratitude to the organizers and community

• Go through the winning solutions

• Add things from the solutions to your toolkit so that your knowledge grows

• Use the knowledge in your work place as well as in future hackathons / competitions

• If possible, do a better writeup of your own solution and share with the community [ Sharing is caring ]

To add a personal touch to the article, let me share something which might inspire people like me who have spent most of their life getting consolation prizes.

After countless attempts in various competitions without success, I finally achieved a major victory one morning. Just a week later, I received another award. Although my streak of participating without wins continues, I do occasionally win awards.

I have been a Kaggler for more than a decade. Though I am a Kaggler, I am not a GrandMaster or anything. Still the passion for looking into competition continues.

This week , Kaggle has made a very nice gesture for all Kagglers and has given badges and award acknowledgments for their contributions to the community.

It feels surreal that the community remembers the efforts

• Conceptual
• Implementation details which comes as expected with code as well as the full code link

Please feel free to choose both or at least the Conceptual section

For this we are using the BBC News Dataset . This is a self updating dataset and is updated daily.

We would be learning Simple and Advanced RAG [ Retrieval Augmented Generation] using a small language model Phi3 mini 128K instruct through this blog.

We would be asking questions like What is the news in Ukraine and the application will provide the answers using this technique.

The Phi-3-Mini-128K-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets. In comparison GPT-4 has more than a trillion parameters and the smallest Llama 3 model has 8 billion. These models such as Phi-3 are popularly known as SLM[ Small Language Models] while the likes of GPT-4, GPT-3.5 Turbo are known as LLM[ Large Language Models]

The concept of Word Embeddings would be widely used in the blog. From the TensorFlow documentation word embeddings documentation

Word embeddings give us a way to use an efficient, dense representation in which similar words have a similar encoding.

Importantly, you do not have to specify this encoding by hand. An embedding is a dense vector of floating point values (the length of the vector is a parameter you specify).

Instead of specifying the values for the embedding manually, they are trainable parameters (weights learned by the model during training, in the same way a model learns weights for a dense layer).

It is common to see word embeddings that are 8-dimensional (for small datasets), up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but takes more data to learn.

RAG has 3 major components

1. Ingestion

2. Querying

3. Generation

Ingestion

For Ingestion, following are the key components

1. Read the Data Source

2. Convert the read text into manageable chunks

3. Convert the manageable chunks into embeddings. This is a technique in which you convert text into an array of numbers

4. Store the embeddings into a vector database

5. Store the metadata such as the filename, text , and other relevant things in the vector database

Query the data using Simple RAG

In the query component, we require 3 main components

1. Orchestrating application which is responsible for coordinating the interactions between the other components such as the user , vector database , Language Model .

2. Vector Database which stores the information

3. Language model which is helpful for generating the information after it has been provided contextual information

Data Flow of a Simple RAG

1. The user inputs the question . Example : What is the news in Ukraine

2. The Orchestrating application uses a encoder to transform the text into embedding We have used the all-MiniLM-L6-v2 of the Sentence Transformer as the encoder

3. The embedding is searched in the Vector database. In this case we have used the Qdrant database as the vector database

4. Search results are obtained from the vector database. We get the top K results from the vector database. The number of results to be obtained is configurable

5. A consolidated answer or popularly called context is prepared from the answers. In the implementation that we would do is done by concatenating the search results

6. This context is sent to the language model for generating the answers relevant for the context. In the implementation we have used a small language model Phi3

Data Flow of a Advanced RAG

The steps remain the same.

Except the following

Step 4 - Search results are obtained from the vector database. We get the top K2 results from the vector database. The number of results to be obtained is configurable. The results K2 is larger than K

Step 4A. The results obtained are passed into a new type of block known as the cross-encoder which distills the number of results and provides a smaller set of results which has high similarity between the results and the query. These smaller set of results can be the top K results.

Implementation details

For this implementation , we have used the following

1. Dataset - BBC News dataset

2. Vector Database - Qdrant. We have a used in memory version of Qdrant for demonstration

3. Language Model - Small language model Phi3

4. Orchestrator application - Kaggle notebook

Setup

Install the python libraries

! pip install -U qdrant-client --quiet
! pip install -U sentence-transformers --quiet

Imports

from qdrant_client import models, QdrantClient
from sentence_transformers import SentenceTransformer,CrossEncoder

Sentence Transformer Encoder

Instantiate the sentence transformer encoder

encoder = SentenceTransformer("all-MiniLM-L6-v2")

Create the Qdrant Collection

We are creating

• In memory qdrant collection

• The collection name is BBC

• The size of the vector embedding to be inserted is the dimention of the encoder . In this case , the dimension when evaluated is 384

• Distance of similarity is cosine

qdrant = QdrantClient(":memory:")

qdrant.recreate_collection(
    collection_name="BBC",
    vectors_config=models.VectorParams(
        size=encoder.get_sentence_embedding_dimension(),  # Vector size is defined by used model
        distance=models.Distance.COSINE,
    ),
)

Data Ingestion

Read the Dataset

Read the BBC News Dataset

LIMIT = 500
df = pd.read_csv("/kaggle/input/bbc-news/bbc_news.csv")
docs = df[:LIMIT]

Upload the documents into Qdrant

import uuid
%%capture --no-display
qdrant.upload_points(
    collection_name="BBC",
    points=[
        models.PointStruct(
            id=str(uuid.uuid4()), 
            vector=encoder.encode(row[1]["title"]),
            payload={ "title":row[1]["title"] ,
                     "description":row[1]["description"] }
        )
        for row in docs.iterrows()
    ],
)

Verify the documents have been uploaded into Qdrant

qdrant.count(
    collection_name="BBC",
    exact=True,
)

If you have reached till this point, Congratulations 👌 . You have been able to complete the understanding of the Data Ingestion into Qdrant

Query the Qdrant database

Query for the user

query_string = "Describe the news for Ukraine"

Search Qdrant for the query

For searching , note how we have converted the user input into a embedding

encoder.encode(query_string).tolist()

hits = qdrant.search(
    collection_name="BBC",
    query_vector=encoder.encode(query_string).tolist(),
    limit=35,
)

for hit in hits:
    print(hit.payload, "score:", hit.score)

Refine the result with the CrossEncoder

We are refining the results from the CrossEncoder .

We have got in our implementation K2 = 35 results from Qdrant. We have used the Cross Encoder cross-encoder/ms-marco-MiniLM-L-6-v2 to refine the results The refined results in our case K = 5 after we pass the results through the cross encoder.

CROSSENCODER_MODEL_NAME = 'cross-encoder/ms-marco-MiniLM-L-6-v2'
RANKER_RESULTS_LIMIT = 5

user_input = query_string

contexts_list = []
for result in hits:
    contexts_list.append(result.payload['description'])

cross_encoder = CrossEncoder(CROSSENCODER_MODEL_NAME)
cross_inp = [[user_input, hit] for hit in contexts_list]
cross_scores = cross_encoder.predict(cross_inp)

cross_scores_text = []
cross_scores_length = len(cross_scores)
for i in range(cross_scores_length):
    d = {}
    d['score'] = cross_scores[i]
    d['text'] = contexts_list[i]
    cross_scores_text.append(d)

hits_selected = sorted(cross_scores_text, key=lambda x: x['score'], reverse=True)
contexts =""
hits = hits_selected[:RANKER_RESULTS_LIMIT]

Create the context

We create the Context for RAG using the search results

contexts =""
for i in range(len(hits)):
    contexts  +=  hits[i]['text']+"\n---\n"

If you have reached till this point, Congratulations 👌 👌 again. You have been able to complete the understanding of the Getting Results from Qdrant [ Vector Database ]

Generate the answer with the Small Language Model

We also use the small language model microsoft/Phi-3-mini-128k-instruct model .

From the Hugging Face model card

The Phi-3-Mini-128K-Instruct is a 3.8 billion-parameter, lightweight, state-of-the-art open model trained using the Phi-3 datasets. This dataset includes both synthetic data and filtered publicly available website data, with an emphasis on high-quality and reasoning-dense properties. The model belongs to the Phi-3 family with the Mini version in two variants 4K and 128K which is the context length (in tokens) that it can support.

From the Microsoft blog

Thanks to their smaller size, Phi-3 models can be used in compute-limited inference environments. Phi-3-mini, in particular, can be used on-device, especially when further optimized with ONNX Runtime for cross-platform availability. The smaller size of Phi-3 models also makes fine-tuning or customization easier and more affordable. In addition, their lower computational needs make them a lower cost option with much better latency. The longer context window enables taking in and reasoning over large text content—documents, web pages, code, and more. Phi-3-mini demonstrates strong reasoning and logic capabilities, making it a good candidate for analytical tasks.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

torch.random.manual_seed(0)

model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-128k-instruct", 
    device_map="cuda", 
    torch_dtype="auto", 
    trust_remote_code=True, 
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-128k-instruct")

Create the prompt

The prompt is created with 2 components

• Context which we created in the section Create the context
• User input which is the user input

prompt = f"""Answer based on context:\n\n{contexts}\n\n{user_input}"""

Create the message template

messages = [
     {"role": "user", "content": prompt},
]

Generate the message

%%time
model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
model_inputs =  model_inputs.to('cuda')
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
decoded = tokenizer.batch_decode(generated_ids)

Print the answer

print(decoded[0].split("<|assistant|>")[-1].split("<|end|>")[0])

Code

The code can be found in the Kaggle notebook BBC NEWS Advanced RAG PHI3

The session gives an overview of Azure AI and the position of Azure OpenAI in the Azure AI ecosystem. We focus on a question and answering system and show the different components of the built question answering system

✅ Header

✅ Context

✅ Prompt

We also have a sneak peak on prompt engineering and delve in the details of different types of models

⭐ text davinci

⭐ gpt 3.5

⭐ gpt 4

We also look deeper in the code and the difference of implementations of the various models