notebooks/content_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extract Content from Your File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrate you can use Content Understanding API to extract semantic content from multimodal files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "1. Ensure Azure AI service is configured following [steps](../README.md#configure-azure-ai-service-resource)\n",
    "2. Install the required packages to run the sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -r ../requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Azure AI Content Understanding Client\n",
    "\n",
    "> The [AzureContentUnderstandingClient](../python/content_understanding_client.py) is a utility class containing functions to interact with the Content Understanding API. Before the official release of the Content Understanding SDK, it can be regarded as a lightweight SDK.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import json\n",
    "import os\n",
    "import sys\n",
    "import uuid\n",
    "from pathlib import Path\n",
    "from dotenv import find_dotenv, load_dotenv\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "\n",
    "load_dotenv(find_dotenv())\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "\n",
    "AZURE_AI_ENDPOINT = os.getenv(\"AZURE_AI_ENDPOINT\")\n",
    "AZURE_AI_API_VERSION = os.getenv(\"AZURE_AI_API_VERSION\", \"2024-12-01-preview\")\n",
    "\n",
    "# Add the parent directory to the path to use shared modules\n",
    "parent_dir = Path(Path.cwd()).parent\n",
    "sys.path.append(str(parent_dir))\n",
    "from python.content_understanding_client import AzureContentUnderstandingClient\n",
    "\n",
    "credential = DefaultAzureCredential()\n",
    "token_provider = get_bearer_token_provider(credential, \"https://cognitiveservices.azure.com/.default\")\n",
    "\n",
    "client = AzureContentUnderstandingClient(\n",
    "    endpoint=AZURE_AI_ENDPOINT,\n",
    "    api_version=AZURE_AI_API_VERSION,\n",
    "    token_provider=token_provider,\n",
    "    x_ms_useragent=\"azure-ai-content-understanding-python/content_extraction\", # This header is used for sample usage telemetry, please comment out this line if you want to opt out.\n",
    ")\n",
    "\n",
    "# Utility function to save images\n",
    "from PIL import Image\n",
    "from io import BytesIO\n",
    "import re\n",
    "\n",
    "def save_image(image_id: str, response):\n",
    "    raw_image = client.get_image_from_analyze_operation(analyze_response=response,\n",
    "        image_id=image_id\n",
    "    )\n",
    "    image = Image.open(BytesIO(raw_image))\n",
    "    # image.show()\n",
    "    Path(\".cache\").mkdir(exist_ok=True)\n",
    "    image.save(f\".cache/{image_id}.jpg\", \"JPEG\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Content\n",
    "\n",
    "Content Understanding API is designed to extract all textual content from a specified document file. In addition to text extraction, it conducts a comprehensive layout analysis to identify and categorize tables and figures within the document. The output is then presented in a structured markdown format, ensuring clarity and ease of interpretation.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANALYZER_ID = \"content-doc-sample-\" + str(uuid.uuid4())\n",
    "ANALYZER_TEMPLATE_FILE = '../analyzer_templates/content_document.json'\n",
    "ANALYZER_SAMPLE_FILE = '../data/invoice.pdf'\n",
    "\n",
    "# Create analyzer\n",
    "response = client.begin_create_analyzer(ANALYZER_ID, analyzer_template_path=ANALYZER_TEMPLATE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "# Analyzer file\n",
    "response = client.begin_analyze(ANALYZER_ID, file_location=ANALYZER_SAMPLE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "print(json.dumps(result, indent=2))\n",
    "client.delete_analyzer(ANALYZER_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Audio Content\n",
    "Our API output facilitates detailed analysis of spoken language, allowing developers to utilize the data for various applications, such as voice recognition, customer service analytics, and conversational AI. The structure of the output makes it easy to extract and analyze different components of the conversation for further processing or insights.\n",
    "\n",
    "1. Speaker Identification: Each phrase is attributed to a specific speaker (in this case, \"Speaker 2\"). This allows for clarity in conversations with multiple participants.\n",
    "1. Timing Information: Each transcription includes precise timing data:\n",
    "    - startTimeMs: The time (in milliseconds) when the phrase begins.\n",
    "    - endTimeMs: The time (in milliseconds) when the phrase ends.\n",
    "    This information is crucial for applications like video subtitles, allowing synchronization between the audio and the text.\n",
    "1. Text Content: The actual spoken text is provided, which in this instance is \"Thank you for calling Woodgrove Travel.\" This is the main content of the transcription.\n",
    "1. Confidence Score: Each transcription phrase includes a confidence score (0.933 in this case), indicating the likelihood that the transcription is accurate. A higher score suggests greater reliability.\n",
    "1. Word-Level Breakdown: The transcription is further broken down into individual words, each with its own timing information. This allows for detailed analysis of speech patterns and can be useful for applications such as language processing or speech recognition improvement.\n",
    "1. Locale Specification: The locale is specified as \"en-US,\" indicating that the transcription is in American English. This is important for ensuring that the transcription algorithms account for regional dialects and pronunciations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANALYZER_ID = \"content-audio-sample-\" + str(uuid.uuid4())\n",
    "ANALYZER_TEMPLATE_FILE = '../analyzer_templates/audio_transcription.json'\n",
    "ANALYZER_SAMPLE_FILE = '../data/audio.wav'\n",
    "\n",
    "# Create analyzer\n",
    "response = client.begin_create_analyzer(ANALYZER_ID, analyzer_template_path=ANALYZER_TEMPLATE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "# Analyzer file\n",
    "response = client.begin_analyze(ANALYZER_ID, file_location=ANALYZER_SAMPLE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "print(json.dumps(result, indent=2))\n",
    "client.delete_analyzer(ANALYZER_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video Content\n",
    "Video output provides detailed information about audiovisual content, specifically video shots. Here are the key features it offers:\n",
    "\n",
    "1. Shot Information: Each shot is defined by a start and end time, along with a unique identifier. For example, Shot 0:0.0 to 0:2.800 includes a transcript and key frames.\n",
    "1. Transcript: The API includes a transcript of the audio, formatted in WEBVTT, which allows for easy synchronization with the video. It captures spoken content and specifies the timing of the dialogue.\n",
    "1. Key Frames: It provides a series of key frames (images) that represent important moments in the video shot, allowing users to visualize the content at specific timestamps.\n",
    "1. Description: Each shot is accompanied by a description, providing context about the visuals presented. This helps in understanding the scene or subject matter without watching the video.\n",
    "1. Audio Visual Metadata: Details about the video such as dimensions (width and height), type (audiovisual), and the presence of key frame timestamps are included.\n",
    "1. Transcript Phrases: The output includes specific phrases from the transcript, along with timing and speaker information, enhancing the usability for applications like closed captioning or search functionalities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANALYZER_ID = \"content-video-sample-\" + str(uuid.uuid4())\n",
    "ANALYZER_TEMPLATE_FILE = '../analyzer_templates/content_video.json'\n",
    "ANALYZER_SAMPLE_FILE = '../data/FlightSimulator.mp4'\n",
    "\n",
    "# Create analyzer\n",
    "response = client.begin_create_analyzer(ANALYZER_ID, analyzer_template_path=ANALYZER_TEMPLATE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "# Analyzer file\n",
    "response = client.begin_analyze(ANALYZER_ID, file_location=ANALYZER_SAMPLE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "print(json.dumps(result, indent=2))\n",
    "\n",
    "# Save keyframes (optional)\n",
    "keyframe_ids = set()\n",
    "result_data = result.get(\"result\", {})\n",
    "contents = result_data.get(\"contents\", [])\n",
    "\n",
    "# Iterate over contents to find keyframes if available\n",
    "for content in contents:\n",
    "    # Extract keyframe IDs from \"markdown\" if it exists and is a string\n",
    "    markdown_content = content.get(\"markdown\", \"\")\n",
    "    if isinstance(markdown_content, str):\n",
    "        keyframe_ids.update(re.findall(r\"(keyFrame\\.\\d+)\\.jpg\", markdown_content))\n",
    "\n",
    "# Output the results\n",
    "print(\"Unique Keyframe IDs:\", keyframe_ids)\n",
    "\n",
    "# Save all keyframe images\n",
    "for keyframe_id in keyframe_ids:\n",
    "    save_image(keyframe_id, response)\n",
    "\n",
    "# Delete analyzer\n",
    "client.delete_analyzer(ANALYZER_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Video Content with Face\n",
    "This is a gated feature, please go through process [Azure AI Resource Face Gating](https://learn.microsoft.com/en-us/legal/cognitive-services/computer-vision/limited-access-identity?context=%2Fazure%2Fai-services%2Fcomputer-vision%2Fcontext%2Fcontext#registration-process) Select `[Video Indexer] Facial Identification (1:N or 1:1 matching) to search for a face in a media or entertainment video archive to find a face within a video and generate metadata for media or entertainment use cases only` in the registration form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ANALYZER_ID = \"content-video-face-sample-\" + str(uuid.uuid4())\n",
    "ANALYZER_TEMPLATE_FILE = '../analyzer_templates/face_aware_in_video.json'\n",
    "ANALYZER_SAMPLE_FILE = '../data/FlightSimulator.mp4'\n",
    "\n",
    "# Create analyzer\n",
    "response = client.begin_create_analyzer(ANALYZER_ID, analyzer_template_path=ANALYZER_TEMPLATE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "# Analyzer file\n",
    "response = client.begin_analyze(ANALYZER_ID, file_location=ANALYZER_SAMPLE_FILE)\n",
    "result = client.poll_result(response)\n",
    "\n",
    "print(json.dumps(result, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get and Save Key Frames and Face Thumbnails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize sets for unique face IDs and keyframe IDs\n",
    "face_ids = set()\n",
    "keyframe_ids = set()\n",
    "\n",
    "# Extract unique face IDs safely\n",
    "result_data = result.get(\"result\", {})\n",
    "contents = result_data.get(\"contents\", [])\n",
    "\n",
    "# Iterate over contents to find faces and keyframes if available\n",
    "for content in contents:\n",
    "    # Safely retrieve face IDs if \"faces\" exists and is a list\n",
    "    faces = content.get(\"faces\", [])\n",
    "    if isinstance(faces, list):\n",
    "        for face in faces:\n",
    "            face_id = face.get(\"faceId\")\n",
    "            if face_id:\n",
    "                face_ids.add(f\"face.{face_id}\")\n",
    "\n",
    "    # Extract keyframe IDs from \"markdown\" if it exists and is a string\n",
    "    markdown_content = content.get(\"markdown\", \"\")\n",
    "    if isinstance(markdown_content, str):\n",
    "        keyframe_ids.update(re.findall(r\"(keyFrame\\.\\d+)\\.jpg\", markdown_content))\n",
    "\n",
    "# Output the results\n",
    "print(\"Unique Face IDs:\", face_ids)\n",
    "print(\"Unique Keyframe IDs:\", keyframe_ids)\n",
    "\n",
    "# Save all face images\n",
    "for face_id in face_ids:\n",
    "    save_image(face_id, response)\n",
    "\n",
    "# Save all keyframe images\n",
    "for keyframe_id in keyframe_ids:\n",
    "    save_image(keyframe_id, response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean Up\n",
    "Optionally, delete the sample analyzer from your resource. In typical usage scenarios, you would analyze multiple files using the same analyzer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.delete_analyzer(ANALYZER_ID)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}