Azure Cognitive Search - Vector, Semantic & Hybrid Search

This repo has python code to demonstrate the use of Azure Cognitive Search with Open AI to provide vector, semantic, and hybrid search capabilities. The code is based on the Azure Cognitive Search - Vector Search (Preview).

Infrastructure Setup

Clone the repo and create a virtual environment for the python scripts.

git clone https://github.com/fsaleemm/acs-vector-semantic.git
cd acs-vector-semantic

Deploy the following Cognitive Services:

1. Azure Open AI
2. Cognitive Search Service

$LOCATION="eastus"
az deployment sub create -n "acs-semantic-vector" -l $LOCATION -f infra/main.bicep -p infra/main.parameters.jsonc -p environmentName="semanticvectorsrc"

This will run for several minutes. Once the deployment is complete, you can get the values for the following resources from the Azure Portal:

1. Azure Search Service Endpoint
2. Azure Search Service Admin Key
3. Open AI Service Endpoint
4. Open AI Service Key

Environment Setup

1. Rename .env-sample to .env and fill in the information from the deployed resources.

    AZURE_SEARCH_SERVICE_ENDPOINT=<Your-Search-Service-Endpoint>
    AZURE_SEARCH_INDEX_NAME=semantic-vector
    AZURE_SEARCH_ADMIN_KEY=<Your-Search-Service-Admin-Key>
    OPENAI_ENDPOINT=<Your-Open-AI-Service-Endpoint>
    OPENAI_API_KEY=<Your-Open-AI-Service-Key>
    OPENAI_API_VERSION=2023-05-15
    OPENAI_EMBEDDING_DEPLOYED_MODEL=acs-emb-ada-002
   

2. Install the required packages

   pip install -r requirements.txt

Search Setup & Vector Data Indexing

1. Run the index creation script

   python .\create-index.py

2. Run the Get Vector script

   python .\get-vector-data.py

3. Run the index vector data script

   python .\index-vector-data.py

Test Search Features

You can run the scripts to see the results of the vector, hybrid, and semantic search features.

1. Run the search vector script

   python .\search-vector.py

   Response Output:

   Score: 0.8404397
   Title: Azure Service Bus
   Content: Azure Service Bus is a fully managed, enterprise-grade messaging service that enables you to build reliable and scalable applications. It provides features like message queuing, publish-subscribe, and dead-lettering. Service Bus supports various messaging patterns, including point-to-point, broadcast, and request-reply. You can use Service Bus to integrate your applications and services, decouple your system components, and handle asynchronous communication. It also integrates with other Azure services, such as Azure Functions and Azure Logic Apps.
   Category: Integration
   
   Score: 0.8315571
   Title: Azure SignalR Service
   Content: Azure SignalR Service is a fully managed, real-time messaging service that enables you to build and scale real-time web applications. It provides features like automatic scaling, WebSocket support, and serverless integration. SignalR Service supports various programming languages, such as C#, JavaScript, and Java. You can use Azure SignalR Service to build chat applications, real-time dashboards, and collaborative tools. It also integrates with other Azure services, such as Azure Functions and Azure App Service.
   Category: Web
   
   Score: 0.82272965
   Title: Azure Queue Storage
   Content: Azure Queue Storage is a fully managed, scalable, and durable message queuing service that enables you to decouple your applications and build asynchronous solutions. It provides features like at-least-once delivery, message time-to-live, and a RESTful API. Queue Storage supports various programming languages, such as C#, Java, and Python. You can use Azure Queue Storage to build distributed applications, offload your databases, and process and store large volumes of messages. It also integrates with other Azure services, such as Azure Functions and Azure Logic Apps.
   Category: Storage
   
   Score: 0.82028526
   Title: Azure Web PubSub
   Content: Azure Web PubSub is a fully managed, real-time messaging service that enables you to build and scale real-time web applications using WebSockets. It provides features like automatic scaling, custom domains, and serverless integration. Web PubSub supports various programming languages, such as C#, JavaScript, and Java. You can use Azure Web PubSub to build chat applications, real-time dashboards, and collaborative tools. It also integrates with other Azure services, such as Azure Functions and Azure SignalR Service.
   Category: Web
   

Troubleshooting

1. If you run into pip install issues, try the following:

   pip install --upgrade pip

2. If you run into issues with Open AI dependencies issue MissingDependencyError, try the following after deactivating the virtual environment:

   #Install the missing dependencies
   pip install openai[datalib]
   
   #Install the required packages
   pip install -r requirements.txt.

Disclaimer

This project is for demonstration purposes only. It is not intended for production use. It is provided without guarantee or support. Use at your own risk.