diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index c7e5c1231..f1cefd9b2 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -20,4 +20,4 @@ This code of conduct applies to all spaces provided by the OpenSource project in **Enforcement and Reporting Code of Conduct Issues:** -Instances of abusive, harassing, or otherwise unacceptable behavior may be reported. [Contact us](mailto:opensource-codeofconduct@amazon.com). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. +Instances of abusive, harassing, or otherwise unacceptable behavior may be reported. [Contact us](mailto:conduct@opensearch.foundation). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. diff --git a/_community_members/nathhjo.md b/_community_members/nathhjo.md new file mode 100644 index 000000000..4b6159860 --- /dev/null +++ b/_community_members/nathhjo.md @@ -0,0 +1,24 @@ +--- +name: Nathalie Jonathan +short_name: nathhjo +photo: "/assets/media/community/members/nathhjo.jpg" +title: 'OpenSearch Community Member: Nathalie Jonathan' +primary_title: Nathalie Jonathan +breadcrumbs: + icon: community + items: + - title: Community + url: /community/index.html + - title: Members + url: /community/members/index.html + - title: 'Nathalie Jonathan's Profile' + url: '/community/members/nathalie-jonathan.html' +github: nathaliellenaa +job_title_and_company: 'Software engineer at Amazon Web Services' +personas: + - author +permalink: '/community/members/nathalie-jonathan.html' +redirect_from: '/authors/nathhjo/' +--- + +**Nathalie Jonathan** is a Software Engineer at AWS working on the OpenSearch ML Commons team. \ No newline at end of file diff --git a/_posts/2024-04-30-A-comprehensive-guide-to-setup-and-connect-self-managed-dashboards-with-Amazon-OpenSearch-domain.md b/_posts/2024-04-30-A-comprehensive-guide-to-setup-and-connect-self-managed-dashboards-with-Amazon-OpenSearch-domain.md index b8a418cb3..63ab9e5c9 100644 --- a/_posts/2024-04-30-A-comprehensive-guide-to-setup-and-connect-self-managed-dashboards-with-Amazon-OpenSearch-domain.md +++ b/_posts/2024-04-30-A-comprehensive-guide-to-setup-and-connect-self-managed-dashboards-with-Amazon-OpenSearch-domain.md @@ -12,6 +12,9 @@ has_math: true has_science_table: true --- +> [!NOTE] +> Before you proceed, Amazon OpenSearch Service has introduced the next-generation OpenSearch UI designed to address this purpose. To learn more, check out: https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-launches-the-next-generation-opensearch-ui + [OpenSearch](https://opensearch.org/) is a scalable, flexible, and extensible open-source software suite for search, analytics, security monitoring, and observability applications, licensed under Apache 2.0. [OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/) is a powerful and flexible data visualization and exploration platform that enables users to analyze and visualize large volumes of data. It is open-source software that provides a user-friendly interface for creating interactive dashboards, charts, and graphs, allowing users to gain valuable insights from their data. In [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/), a blue/green deployment establishes a standby environment for domain updates by replicating the production environment. After completing the updates, users are directed to the new environment. The blue environment represents the current production setup, while the green environment represents the standby setup. After completing the upgrade process, OpenSearch Service switches the environments, promoting the green environment to become the new production environment without any data loss. However, due to the current code configuration, access to Dashboards is interrupted during the initial phase of blue/green deployment. This can result in downtime for Dashboards, which presents challenges to users because it restricts their ability to visualize and explore data during this period. diff --git a/_posts/2025-01-28-OpenSearch-Now-Supports-DeepSeek-Chat-Models.md b/_posts/2025-01-28-OpenSearch-Now-Supports-DeepSeek-Chat-Models.md new file mode 100644 index 000000000..ae34d9d44 --- /dev/null +++ b/_posts/2025-01-28-OpenSearch-Now-Supports-DeepSeek-Chat-Models.md @@ -0,0 +1,270 @@ +--- +layout: post +title: "OpenSearch now supports DeepSeek chat models" +authors: + - seanzheng + - ylwu + - nathhjo + - kolchfa +date: 2025-01-29 +categories: + - technical-posts +meta_keywords: OpenSearch DeepSeek integration, LLM integration, RAG, AI search, machine learning, natural language processing, open-source LLM +meta_description: Explore how OpenSearch's integration with DeepSeek R1 LLM models enables cost-effective Retrieval-Augmented Generation (RAG) while maintaining high performance comparable to leading LLMs. +--- + +We're excited to announce that OpenSearch now supports DeepSeek integration, providing powerful and cost-effective AI capabilities. DeepSeek R1 is a recently released open-source large language model (LLM) that delivers **similar benchmarking performance** to leading LLMs like OpenAI O1 ([report](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf)) at a significantly **lower cost** ([DeepSeek API pricing](https://api-docs.deepseek.com/quick_start/pricing)). Because DeepSeek R1 is open source, you can download and deploy it to your preferred infrastructure. This enables you to build more cost-effective and sustainable retrieval-augmented generation (RAG) solutions. + +OpenSearch gives you the flexibility to connect to any remote inference service, such as DeepSeek or OpenAI, using machine learning (ML) connectors. You can use [prebuilt connector blueprints](https://github.com/opensearch-project/ml-commons/tree/main/docs/remote_inference_blueprints) or customize connectors based on your requirements. For more information about connector blueprints, see [Blueprints](https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/blueprints/). + +We've added a new [connector blueprint](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/deepseek_connector_chat_blueprint.md) for the DeepSeek R1 model. This integration, combined with OpenSearch's built-in vector database capabilities, makes it easier and more cost effective to build [RAG applications](https://opensearch.org/docs/latest/search-plugins/conversational-search) in OpenSearch. + +The following example shows you how to implement RAG with DeepSeek in OpenSearch's vector database. This example guides you through creating a connector for the [DeepSeek chat model](https://api-docs.deepseek.com/api/create-chat-completion) and setting up a [RAG pipeline](https://opensearch.org/docs/latest/search-plugins/search-pipelines/rag-processor/) in OpenSearch. + +### 1. Create a connector for DeepSeek + +First, create a connector for the DeepSeek chat model, providing your own DeepSeek API key: + +```json +POST /_plugins/_ml/connectors/_create +{ + "name": "DeepSeek Chat", + "description": "Test connector for DeepSeek Chat", + "version": "1", + "protocol": "http", + "parameters": { + "endpoint": "api.deepseek.com", + "model": "deepseek-chat" + }, + "credential": { + "deepSeek_key": "" + }, + "actions": [ + { + "action_type": "predict", + "method": "POST", + "url": "https://${parameters.endpoint}/v1/chat/completions", + "headers": { + "Content-Type": "application/json", + "Authorization": "Bearer ${credential.deepSeek_key}" + }, + "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }" + } + ] +} +``` + +The response contains a connector ID for the newly created connector: + +```json +{ + "connector_id": "n0dOqZQBQwAL8-GO1pYI" +} +``` + +For more information, see [Connecting to externally hosted models](https://opensearch.org/docs/latest/ml-commons-plugin/remote-models/index/). + +### 2. Create a model group + +Create a model group for the DeepSeek chat model: + +```json +POST /_plugins/_ml/model_groups/_register +{ + "name": "remote_model_group_chat", + "description": "This is an example description" +} +``` + +The response contains a model group ID: + +```json +{ + "model_group_id": "b0cjqZQBQwAL8-GOVJZ4", + "status": "CREATED" +} +``` + +For more information about model groups, see [Model access control](https://opensearch.org/docs/latest/ml-commons-plugin/model-access-control/). + +### 3. Register and deploy the model + +Register the model to the model group and deploy the model using the model group ID and connector ID created in the previous steps: + +```json +POST /_plugins/_ml/models/_register?deploy=true +{ + "name": "DeepSeek Chat model", + "function_name": "remote", + "model_group_id": "b0cjqZQBQwAL8-GOVJZ4", + "description": "DeepSeek Chat", + "connector_id": "n0dOqZQBQwAL8-GO1pYI" +} +``` + +The response contains the model ID: + +```json +{ + "task_id": "oEdPqZQBQwAL8-GOCJbw", + "status": "CREATED", + "model_id": "oUdPqZQBQwAL8-GOCZYL" +} +``` + +To ensure that the connector is working as expected, test the model: + +```json +POST /_plugins/_ml/models/oUdPqZQBQwAL8-GOCZYL/_predict +{ + "parameters": { + "messages": [ + { + "role": "system", + "content": "You are a helpful assistant." + }, + { + "role": "user", + "content": "Hello!" + } + ] + }, + "stream": false +} +``` + +The response verifies that the connector is working as expected: + +```json +{ + "inference_results": [ + { + "output": [ + { + "name": "response", + "dataAsMap": { + "id": "9d9bd689-88a5-44b0-b73f-2daa92518761", + "object": "chat.completion", + "created": 1.738011126E9, + "model": "deepseek-chat", + "choices": [ + { + "index": 0.0, + "message": { + "role": "assistant", + "content": "Hello! How can I assist you today? 😊" + }, + "finish_reason": "stop" + } + ], + "usage": { + "prompt_tokens": 11.0, + "completion_tokens": 11.0, + "total_tokens": 22.0, + "prompt_tokens_details": { + "cached_tokens": 0.0 + }, + "prompt_cache_hit_tokens": 0.0, + "prompt_cache_miss_tokens": 11.0 + }, + "system_fingerprint": "fp_3a5770e1b4" + } + } + ], + "status_code": 200 + } + ] +} +``` + +### 4. Create a search pipeline + +Create a search pipeline with a `retrieval_augmented_generation` processor: + +```json +PUT /_search/pipeline/rag_pipeline +{ + "response_processors": [ + { + "retrieval_augmented_generation": { + "tag": "openai_pipeline_demo", + "description": "Demo pipeline Using OpenAI Connector", + "model_id": "oUdPqZQBQwAL8-GOCZY", + "context_field_list": ["text"], + "system_prompt": "You are a helpful assistant", + "user_instructions": "Generate a concise and informative answer in less than 100 words for the given question" + } + } + ] +} +``` + +For more information, see [Conversational search](https://opensearch.org/docs/latest/search-plugins/conversational-search). + +### 5. Create a conversation memory + +Assuming that you created a k-NN index and ingested the data, you can now create a conversation memory. For more information about creating a k-NN index, see [k-NN index](https://opensearch.org/docs/latest/search-plugins/knn/knn-index/). For more information about ingesting data, see [Ingest RAG data into an index](https://opensearch.org/docs/latest/search-plugins/conversational-search/#step-4-ingest-rag-data-into-an-index). + +Create a conversation memory to store all messages from a conversation: + +```json +POST /_plugins/_ml/memory/ +{ + "name": "Conversation about NYC population" +} +``` + +The response contains a memory ID for the created memory: + +```json +{ + "memory_id": "znCqcI0BfUsSoeNTntd7" +} +``` + +### 6. Use the pipeline for RAG + +Send a query to OpenSearch and provide additional parameters in the `ext.generative_qa_parameters` object: + +```json +GET /my_rag_test_data/_search +{ + "query": { + "match": { + "text": "What's the population of NYC metro area in 2023" + } + }, + "ext": { + "generative_qa_parameters": { + "llm_model": "gpt-3.5-turbo", + "llm_question": "What's the population of NYC metro area in 2023", + "memory_id": "znCqcI0BfUsSoeNTntd7", + "context_size": 5, + "message_size": 5, + "timeout": 15 + } + } +} +``` + +The response contains the model output: + +```json +{ + ... + "ext": { + "retrieval_augmented_generation": { + "answer": "The population of the New York City metro area in 2022 was 18,867,000.", + "message_id": "p3CvcI0BfUsSoeNTj9iH" + } + } +} +``` + +## Wrapping up + +By integrating DeepSeek R1, OpenSearch continues its mission to democratize AI-powered search and analytics—offering developers **more choice, greater flexibility, and lower costs**. + +**Try DeepSeek R1 now!** + +As always, we welcome your feedback, and we'd love to hear from you on the [OpenSearch forum](https://forum.opensearch.org/). \ No newline at end of file diff --git a/assets/media/community/members/nathhjo.jpg b/assets/media/community/members/nathhjo.jpg new file mode 100644 index 000000000..f3c08fa76 Binary files /dev/null and b/assets/media/community/members/nathhjo.jpg differ diff --git a/release-dashboard/index.md b/release-dashboard/index.md index b276a6815..72a194ade 100644 --- a/release-dashboard/index.md +++ b/release-dashboard/index.md @@ -12,6 +12,9 @@ breadcrumbs: omit_from_search: true release_versions: + - version: 2.19.0 + release_issue: 5152 + release_retro_issue: 5153 - version: 3.0.0 release_issue: 3747 release_retro_issue: 5174 @@ -328,4 +331,4 @@ metrics_height_mobile: 6000

To learn more about OpenSearch Release Process, please read this wiki document part of the opensearch-build repo. -

\ No newline at end of file +

diff --git a/releases.md b/releases.md index 3b2bde01b..c2d55faee 100644 --- a/releases.md +++ b/releases.md @@ -42,7 +42,7 @@ Note: We have not added a major release to the 2024 schedule yet. If/when we a | 2.17.1 | September 24th, 2024 | October 1st, 2024 | [Divya Madala](https://github.com/Divyaasm/) | [5046](https://github.com/opensearch-project/opensearch-build/issues/5046) | | 2.18.0 | October 22nd, 2024 | November 05th, 2024 | [Rishabh Singh](https://github.com/rishabh6788/) | [5004](https://github.com/opensearch-project/opensearch-build/issues/5004) | | 1.3.20 | December 03rd, 2024 | December ~~10th~~ 11th, 2024 | [Brandon Shien](https://github.com/bshien) | [4990](https://github.com/opensearch-project/opensearch-build/issues/4990) | -| 2.19.0 | January 28th, 2025 | Feb 11th, 2025 | | | +| 2.19.0 | January 28th, 2025 | Feb 11th, 2025 | [Rishabh Singh](https://github.com/rishabh6788) | [5152](https://github.com/opensearch-project/opensearch-build/issues/5152) | {: .desktop-release-schedule-table} OpenSearch [follows semver](https://opensearch.org/blog/technical-post/2021/08/what-is-semver/), which means we will only release breaking changes in major versions. All minor versions are compatible with every other minor version for that major. For example, 1.2.0 will work with 1.3.2, 1.4.1, etc, but may not work with 2.0.