Merge pull request AzureCosmosDB#11 from solliancenet/add-guide

Merge DiskANN dev guide into the repo

README.md

@@ -1,21 +1,11 @@
 # Azure Cosmos DB + Azure OpenAI Python Developer Guide
 
-1. [Introduction](/00_Introduction/README.md)
-1. [Azure Overview](/01_Azure_Overview/README.md)
-1. [Overview of Azure Cosmos DB](/02_Overview_Cosmos_DB/README.md)
-1. [Overview of Azure OpenAI](/03_Overview_Azure_OpenAI/README.md)
-1. [Overview of AI Concepts](/04_Overview_AI_Concepts/README.md)
-1. [Explore the Azure OpenAI models and endpoints (console app)](/05_Explore_OpenAI_models/README.md)
-1. [Provision Azure resources](/06_Provision_Azure_Resources/README.md)
-1. [Create your first Cosmos DB project](/07_Create_First_Cosmos_DB_Project/README.md)
-1. [Load data into Azure Cosmos DB API for MongoDB](/08_Load_Data/README.md)
-1. [Use vector search on embeddings in vCore-based Azure Cosmos DB for MongoDB](/09_Vector_Search_Cosmos_DB/README.md)
-1. [LangChain](/10_LangChain/README.md)
-1. [Backend API](/11_Backend_API/README.md)
-1. [Connect the chat user interface with the chatbot API](/12_User_Interface/README.md)
-1. [Conclusion](/13_Conclusion/README.md)
-
-![Azure Cosmos DB + Azure OpenAI Python Developer Guide Architecture Diagram](/06_Provision_Azure_Resources/media/architecture.jpg)
+Choose one of the following developer guides:
+
+- [Azure Cosmos DB for MongoDB (vCore) + Azure OpenAI Python Developer Guide
+](/vcore/README.md)
+- [Azure Cosmos DB for NoSQL + DiskANN + Azure OpenAI Python Developer Guide
+](/diskann/README.md)
 
 ## Contributing
 

diskann/00_Introduction/README.md

@@ -0,0 +1,21 @@
+# Welcome to the Azure Cosmos DB for NoSQL + DiskANN + Azure OpenAI Python Developer Guide
+
+## Pre-requisites
+
+- [Azure account and subscription](https://azure.microsoft.com/free/) with Owner permissions
+- [Python 3.11 or higher](https://www.python.org/downloads/)
+- [Visual Studio Code](https://code.visualstudio.com/download)
+- [Python extension for VS Code](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
+- [Jupyter Notebook extension for VS Code](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter)
+- [Docker Desktop](https://www.docker.com/products/docker-desktop/) with [WSL 2 backend (if on Windows)](https://learn.docker.com/desktop/wsl/)
+- [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli)
+- [Bicep CLI](https://learn.microsoft.com/azure/azure-resource-manager/bicep/install#install-manually)
+- [Powershell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell?view=powershell-7.3)
+
+## Why use this guide?
+
+The future of software involves combining AI and data services, also known as intelligent applications. This guide is for developers looking to implement intelligent applications quickly while leveraging existing skills. The content will focus on the developer journey implementing an Azure-based AI-enabled GPT-based chat application that is augmented using data stored in Azure Cosmos DB for NoSQL while leveraging Azure OpenAI services.
+
+## Introduction
+
+This guide will walks through the creating intelligent solutions that combines Azure Cosmos DB for NoSQL with vector search capabilities powered by DiskANN and document retrieval with Azure OpenAI services to build a chat bot experience. The guide includes labs that build and deploy a sample chat app using these technologies, with a focus on Azure Cosmos DB for NoSQL, vector search powered by DiskANN, and Azure OpenAI using the Python programming language. For those new to using Azure OpenAI and Vector Search technologies, the guide includes explanations of the core concepts and techniques used when implementing these technologies.

diskann/01_Azure_Overview/README.md

@@ -0,0 +1,150 @@
+# Azure Overview
+
+Millions of customers worldwide trust the Azure platform, and there are over 90,000 Cloud Solution Providers (CSPs) partnered with Microsoft to add extra benefits and services to the Azure platform. By leveraging Azure, organizations can easily modernize their applications, expedite application development, and adapt application requirements to meet the demands of their users. This section provides an overview of Azure, its services, and recommendations on how to get started.
+
+## Advantages of choosing Azure
+
+By offering solutions on Azure, ISVs can access one of the largest B2B markets in the world. Through the [Azure Partner Builder's Program](https://partner.microsoft.com/marketing/azure-isv-technology-partners), Microsoft assists ISVs with the tools and platform to offer their solutions for customers to evaluate, purchase, and deploy with just a few clicks of the mouse.
+
+One of the advantages of choosing Microsoft Azure is access to the Azure AI Services and [Azure OpenAI Service](https://azure.microsoft.com/products/ai-services/openai-service), which empowers developers to integrate advanced artificial intelligence and natural language processing capabilities into their solutions. Developers are able to build, deploy, and manage applications with the language or platform of their choice. With Azure OpenAI Service and other Azure AI services, AI is now available to developers of all skill levels to build scale without constraint.
+
+Microsoft's development suite includes such tools as the various [Visual Studio](https://visualstudio.microsoft.com/) products, [Azure DevOps](https://dev.azure.com/), [GitHub](https://github.com/), and low-code [Power Apps](https://powerapps.microsoft.com/). All of these contribute to Azure's success and growth through their tight integrations with the Azure platform. Organizations that adopt modern tools are 65% more innovative, according to a [2020 McKinsey & Company report.](https://azure.microsoft.com/mediahandler/files/resourcefiles/developer-velocity-how-software-excellence-fuels-business-performance/Developer-Velocity-How-software-excellence-fuels-business-performance-v4.pdf)
+
+![This image demonstrates common development tools on the Microsoft cloud platform to expedite application development.](media/ISV-Tech-Builders-tools-white.png "Microsoft cloud tooling")
+
+To facilitate developers' adoption of Azure, Microsoft offers a [free subscription](https://azure.microsoft.com/free/search/) with $200 credit, applicable for thirty days; year-long access to free quotas for popular services and access to always free Azure service tiers.
+
+## Introduction to Azure resource management
+
+The [Azure Fundamentals Microsoft Learn Module](https://learn.microsoft.com/learn/modules/intro-to-azure-fundamentals/) demonstrates the different classifications of Azure Services. Moreover, Azure supports a variety of common tools, such as Visual Studio, PowerShell, and the Azure CLI, to manage Azure environments.
+
+### The Azure resource management hierarchy
+
+Azure provides a flexible resource hierarchy to simplify cost management and security. This hierarchy consists of four levels:
+
+- **[Management groups](https://learn.microsoft.com/azure/governance/management-groups/overview)**: Management groups consolidate multiple Azure subscriptions for compliance and security purposes.
+- **Subscriptions**: Subscriptions govern cost control and access management. Azure users cannot provision Azure resources without a subscription.
+- **[Resource groups](https://learn.microsoft.com/azure/azure-resource-manager/management/manage-resource-groups-portal)**: Resource groups consolidate the individual Azure resources for a given deployment. All provisioned Azure resources belong to one resource group. In this guide, it will be required to provision a *resource group* in an *subscription* to hold the required resources.
+  - Resource groups are placed in a geographic location that determines where metadata about that resource group is stored.
+- **Resources**: An Azure resource is an instance of a service. An Azure resource belongs to one resource group located in one subscription.
+  - Most Azure resources are provisioned in a particular region.
+
+  ![This image shows Azure resource scopes.](media/scope-levels.png "Azure resource scopes")
+
+### Create landing zone
+
+An [Azure landing zone](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/landing-zone/) is the target environment defined as the final resting place of a cloud migration project. In most projects, the landing zone should be scripted via ARM templates for its initial setup. Finally, it should be customized with PowerShell or the Azure Portal to fit the workload's needs.  First-time Azure users will find creating and deploying to DEV and TEST environments easy.
+
+To help organizations quickly move to Azure, Microsoft provides the Azure landing zone accelerator, which generates a landing zone ARM template according to an organization's core needs, governance requirements, and automation setup. The landing zone accelerator is available in the Azure portal.
+
+![This image demonstrates the Azure landing zone accelerator in the Azure portal, and how organizations can optimize Azure for their needs and innovate.](media/landing-zone-accelerator.png "Azure landing zone accelerator screenshot")
+
+### Automating and managing Azure services
+
+When it comes to managing Azure resources, there are many potential options. [Azure Resource Manager](https://learn.microsoft.com/azure/azure-resource-manager/management/overview) is the deployment and management service for Azure. It provides a management layer that enables users to create, update, and delete resources in Azure subscriptions. Use management features, like access control, locks, and tags, to secure and organize resources after deployment.
+
+All Azure management tools, including the [Azure CLI](https://learn.microsoft.com/cli/azure/what-is-azure-cli), [Azure PowerShell](https://learn.microsoft.com/powershell/azure/what-is-azure-powershell?view=azps-7.1.0) module, [Azure REST API](https://learn.microsoft.com/rest/api/azure/), and browser-based Portal, interact with the Azure Resource Manager layer and [Identity and access management (IAM)](https://learn.microsoft.com/azure/role-based-access-control/overview) security controls.
+
+  ![This image demonstrates how the Azure Resource Manager provides a robust, secure interface to Azure resources.](media/consistent-management-layer.png "Azure Resource Manager explained")
+
+Access control to all Azure services is offered via the [Azure role-based access control (Azure RBAC)](https://learn.microsoft.com/azure/role-based-access-control/overview) natively built into the management platform. Azure RBAC is a system that provides fine-grained access management of Azure resources. Using Azure RBAC, it is possible to segregate duties within teams and grant only the amount of access to users that they need to perform their jobs.
+
+### Azure management tools
+
+The flexibility and variety of Azure's management tools make it intuitive for any user, irrespective of their skill level with specific technologies. As an individual's skill level and administration needs mature, Azure has the right tools to match those needs.
+
+![Azure service management tool maturity progression.](media/azure-management-tool-maturity.png "Azure service management tool")
+
+#### Azure portal
+
+As a new Azure user, the first resource a person will be exposed to is the Azure Portal.  The **Azure Portal** gives developers and architects a view of the state of their Azure resources. It supports extensive user configuration and simplifies reporting. The **[Azure mobile app](https://azure.microsoft.com/get-started/azure-portal/mobile-app/)** provides similar features for users that are away from their main desktop or laptop.
+
+  ![The picture shows the initial Azure service list.](media/azure-portal-services.png "Azure portal Services")
+
+Azure runs on a common framework of backend resource services, and every action taken in the Azure portal translates into a call to a set of backend APIs developed by the respective engineering team to read, create, modify, or delete resources.
+
+##### Azure Marketplace
+
+[Azure Marketplace](https://learn.microsoft.com/marketplace/azure-marketplace-overview) is an online store that contains thousands of IT software applications and services built by industry-leading technology companies. In Azure Marketplace, it is possible to find, try, buy, and deploy the software and services needed to build new solutions and manage the cloud infrastructure. The catalog includes solutions for different industries and technical areas, free trials, and consulting services from Microsoft partners.
+
+![The picture shows an example of Azure Marketplace search results.](media/azure-marketplace-search-results.png "Azure Marketplace Results")
+
+##### Evolving
+
+Moving workloads to Azure alleviates some administrative burdens, but not all. Even though there is no need to worry about the data center, there is still a responsibility for service configuration and user access. Applications will need resource authorization.
+
+Using the existing command-line tools and REST APIs, it is possible to build custom tools to automate and report resource configurations that do not meet organizational requirements.
+
+#### Azure PowerShell and CLI
+
+**Azure PowerShell** and the **Azure CLI** (for Bash shell users) are useful for automating tasks that cannot be performed in the Azure portal. Both tools follow an *imperative* approach, meaning that users must explicitly script the creation of resources in the correct order.
+
+  ![Shows an example of the Azure CLI.](media/azure-cli-example.png "Azure CLI Example")
+
+There are subtle differences between how each of these tools operates and the actions that can be accomplished. Use the [Azure command-line tool guide](https://learn.microsoft.com/azure/developer/azure-cli/choose-the-right-azure-command-line-tool) to determine the right tool to meet the target goal.
+
+#### Azure CLI
+
+It is possible to run the Azure CLI and Azure PowerShell from the [Azure Cloud Shell](https://shell.azure.com), but it does have some limitations. It is also possible to run these tools locally.  
+
+To use the Azure CLI, [download the CLI tools from Microsoft.](https://learn.microsoft.com/cli/azure/install-azure-cli)
+
+To use the Azure PowerShell cmdlets, install the `Az` module from the PowerShell Gallery, as described in the [installation document.](https://learn.microsoft.com/powershell/azure/install-az-ps?view=azps-6.6.0)
+
+##### Azure Cloud Shell
+
+The Azure Cloud Shell provides Bash and PowerShell environments for managing Azure resources imperatively. It also includes standard development tools, like Visual Studio Code, and files are persisted in an Azure Files share.
+
+Launch the Cloud Shell in a browser at [https://shell.azure.com](https://shell.azure.com).
+
+#### PowerShell Module
+
+The Azure portal and Windows PowerShell can be used for managing Azure Cosmos DB for NoSQL. To get started with Azure PowerShell, install the [Azure PowerShell cmdlets](https://learn.microsoft.com/powershell/module/az.cosmosdb/) for Azure Cosmos DB with the following PowerShell command in an administrator-level PowerShell window:
+
+```PowerShell
+Install-Module -Name Az.CosmosDB
+```
+
+#### Infrastructure as Code
+
+[Infrastructure as Code (IaC)](https://learn.microsoft.com/devops/deliver/what-is-infrastructure-as-code) provides a way to describe or declare what infrastructure looks like using descriptive code. The infrastructure code is the desired state. The environment will be built when the code runs and completes. One of the main benefits of IaC is that it is human readable. Once the environment code is proven and tested, it can be versioned and saved into source code control. Developers can review the environment changes over time.
+
+There are a few options of IaC tooling to choose from when provisioning and managing Azure resources. These include Azure-native tools from Microsoft, like ARM templates and Azure Bicep, as well as third-party tools popular within the industry like HashiCorp Terraform.
+
+##### ARM templates
+
+[ARM templates](https://learn.microsoft.com/azure/azure-resource-manager/templates/) can deploy Azure resources in a *declarative* manner. Azure Resource Manager can potentially create the resources in an ARM template in parallel. ARM templates can be used to create multiple identical environments, such as development, staging, and production environments.
+
+  ![The picture shows an example of an ARM template JSON export.](media/azure-template-json-example.png "Azure Template JSON")
+
+##### Bicep
+
+Reading, updating, and managing the ARM template JSON code can be difficult for a reasonably sized environment. What if there was a tool that translates simple declarative statements into ARM templates? Better yet, what if there was a tool that took existing ARM templates and translated them into a simple configuration? [Bicep](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview) is a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. Bicep files define the infrastructure to deploy to Azure and then use that file throughout the development lifecycle to repeatedly deploy infrastructure changes. Resources are deployed consistently.
+
+By using the Azure CLI it is possible to decompile ARM templates to Bicep using the following:
+
+```powershell
+az bicep decompile --file template.json
+```
+
+Additionally, the [Bicep playground](https://aka.ms/bicepdemo) tool can perform similar decompilation of ARM templates.
+
+![Sample Bicep code that deploys Azure Cosmos DB for NoSQL](media/bicep_code.png)
+
+##### Terraform
+
+[Hashicorp Terraform](https://www.terraform.io/) is an open-source tool for provisioning and managing cloud infrastructure resources. [Terraform](https://learn.microsoft.com/azure/developer/terraform/overview) simplifies the deployment of Azure services, including Azure Kubernetes Service, Azure Cosmos DB, and Azure AI, through infrastructure-as-code to automate provisioning and management of Azure services. Terraform is also adept at deploying infrastructure across multiple cloud providers. It enables developers to use consistent tooling to manage each infrastructure definition.
+
+![Sample Terraform code that deploys Azure Cosmos DB for NoSQL](media/terraform_code.png)
+
+#### Other tips
+
+Azure administrators should consult with cloud architects and financial and security personnel to develop an effective organizational hierarchy of resources.
+
+Here are some best practices to follow for Azure deployments.
+
+- **Utilize Management Groups** Create at least three levels of management groups.
+
+- **Adopt a naming convention:** Azure resource names should include business details, such as the organization department, and operational details for IT personnel, like the workload. Defining an [Azure resource naming convention](https://learn.microsoft.com/azure/cloud-adoption-framework/ready/azure-best-practices/resource-naming) will help the organization standardize on a common naming convention that will help better identify resources once created.
+
+- **Adopt other Azure governance tools:** Azure provides mechanisms such as [resource tags](https://learn.microsoft.com/azure/azure-resource-manager/management/tag-resources?tabs=json) and [resource locks](https://learn.microsoft.com/azure/azure-resource-manager/management/lock-resources?tabs=json) to facilitate compliance, cost management, and security.