Skip to content

Commit 1e22bdc

Browse files
RuellePaulpaulruelle
RuellePaul
and
paulruelle
authored
feat: lab 3307 aau i use the kilillm functions to import llm data static (#1841)
Co-authored-by: paulruelle <[email protected]>
1 parent 423b2db commit 1e22bdc

26 files changed

+3363
-1298
lines changed

‎docs/sdk/tutorials/llm_project_setup.md renamed to ‎docs/sdk/tutorials/llm_dynamic.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<!-- FILE AUTO GENERATED BY docs/utils.py DO NOT EDIT DIRECTLY -->
2-
<a href="https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_project_setup.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
2+
<a href="https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_dynamic.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
33

44
# How to Set Up a Kili Project with a LLM Model and Create a Conversation
55

@@ -71,7 +71,7 @@ kili = Kili(
7171
# api_endpoint="https://cloud.kili-technology.com/api/label/v2/graphql",
7272
)
7373
project = kili.create_project(
74-
title="[Kili SDK Notebook]: LLM Project",
74+
title="[Kili SDK Notebook]: LLM Dynamic",
7575
description="Project Description",
7676
input_type="LLM_INSTR_FOLLOWING",
7777
json_interface=interface,

‎docs/sdk/tutorials/llm_static.md

+219
Large diffs are not rendered by default.

‎docs/tutorials.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,9 @@ Webhooks are really similar to plugins, except they are self-hosted, and require
7373

7474
## LLM
7575

76-
[This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/llm_project_setup/) will show you how to set up a Kili project that uses a Large Language Model (LLM), create and associate the LLM model with the project, and initiate a conversation using the Kili Python SDK.
76+
[This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/llm_static/) explains how to import conversations into a Kili project to annotate responses generated by a Large Language Model (LLM).
77+
78+
[This tutorial](https://python-sdk-docs.kili-technology.com/latest/sdk/tutorials/llm_dynamic/) guides you through setting up a Kili project with an integrated LLM. You'll learn how to create and link the LLM model to the project and initiate a conversation using the Kili SDK.
7779

7880

7981
## Integrations

‎mkdocs.yml

+3-1
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,9 @@ nav:
6060
- Exporting Project Data:
6161
- Exporting a Project: sdk/tutorials/export_a_kili_project.md
6262
- Parsing Labels: sdk/tutorials/label_parsing.md
63-
- LLM Projects: sdk/tutorials/llm_project_setup.md
63+
- LLM Projects:
64+
- Importing Conversations: sdk/tutorials/llm_static.md
65+
- Model Configuration: sdk/tutorials/llm_dynamic.md
6466
- Setting Up Plugins:
6567
- Developing Plugins: sdk/tutorials/plugins_development.md
6668
- Plugin Example - Programmatic QA: sdk/tutorials/plugins_example.md

‎recipes/img/llm_conversations.png

226 KB
Loading

‎recipes/llm_project_setup.ipynb renamed to ‎recipes/llm_dynamic.ipynb

+2-2
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
"cell_type": "markdown",
55
"metadata": {},
66
"source": [
7-
"<a href=\"https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_project_setup.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
7+
"<a href=\"https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_dynamic.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
88
]
99
},
1010
{
@@ -110,7 +110,7 @@
110110
" # api_endpoint=\"https://cloud.kili-technology.com/api/label/v2/graphql\",\n",
111111
")\n",
112112
"project = kili.create_project(\n",
113-
" title=\"[Kili SDK Notebook]: LLM Project\",\n",
113+
" title=\"[Kili SDK Notebook]: LLM Dynamic\",\n",
114114
" description=\"Project Description\",\n",
115115
" input_type=\"LLM_INSTR_FOLLOWING\",\n",
116116
" json_interface=interface,\n",

‎recipes/llm_static.ipynb

+294
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,294 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"<a href=\"https://colab.research.google.com/github/kili-technology/kili-python-sdk/blob/main/recipes/llm_static.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
8+
]
9+
},
10+
{
11+
"cell_type": "markdown",
12+
"metadata": {},
13+
"source": [
14+
"# How to Set Up a Kili LLM Static project"
15+
]
16+
},
17+
{
18+
"cell_type": "markdown",
19+
"metadata": {},
20+
"source": [
21+
"In this tutorial you'll learn how to create and import conversations in a Kili project with a custom interface for comparing LLM outputs.\n",
22+
"\n",
23+
"\n",
24+
"Here are the steps we will follow:\n",
25+
"\n",
26+
"1. Creating a Kili project with a custom interface\n",
27+
"2. Import three conversations to the project"
28+
]
29+
},
30+
{
31+
"cell_type": "markdown",
32+
"metadata": {},
33+
"source": [
34+
"## Creating a Kili Project with a Custom Interface"
35+
]
36+
},
37+
{
38+
"cell_type": "markdown",
39+
"metadata": {},
40+
"source": [
41+
"We will create a Kili project with a custom interface that includes several jobs for comparing LLM outputs.\n",
42+
"\n",
43+
"### Defining Three Levels of Annotation Jobs\n",
44+
"\n",
45+
"To streamline the annotation process, we define three distinct levels of annotation jobs:\n",
46+
"\n",
47+
"- **Completion:** This job enables annotators to evaluate individual responses generated by LLMs. Each response is annotated separately.\n",
48+
"\n",
49+
"- **Round:** This job allows annotators to assess a single round of conversation, grouping all the LLM responses within that round under a single annotation.\n",
50+
"\n",
51+
"- **Conversation:** This job facilitates annotation at the conversation level, where the entire exchange can be evaluated as a whole.\n",
52+
"\n",
53+
"In this example, we use a JSON interface that incorporates classifications at all these levels, enabling comprehensive annotation:"
54+
]
55+
},
56+
{
57+
"cell_type": "code",
58+
"execution_count": null,
59+
"metadata": {},
60+
"outputs": [],
61+
"source": [
62+
"interface = {\n",
63+
" \"jobs\": {\n",
64+
" \"CLASSIFICATION_JOB_AT_COMPLETION_LEVEL\": {\n",
65+
" \"content\": {\n",
66+
" \"categories\": {\n",
67+
" \"TOO_SHORT\": {\"children\": [], \"name\": \"Too short\", \"id\": \"category1\"},\n",
68+
" \"JUST_RIGHT\": {\"children\": [], \"name\": \"Just right\", \"id\": \"category2\"},\n",
69+
" \"TOO_VERBOSE\": {\"children\": [], \"name\": \"Too verbose\", \"id\": \"category3\"},\n",
70+
" },\n",
71+
" \"input\": \"radio\",\n",
72+
" },\n",
73+
" \"instruction\": \"Verbosity\",\n",
74+
" \"level\": \"completion\",\n",
75+
" \"mlTask\": \"CLASSIFICATION\",\n",
76+
" \"required\": 0,\n",
77+
" \"isChild\": False,\n",
78+
" \"isNew\": False,\n",
79+
" },\n",
80+
" \"CLASSIFICATION_JOB_AT_COMPLETION_LEVEL_1\": {\n",
81+
" \"content\": {\n",
82+
" \"categories\": {\n",
83+
" \"NO_ISSUES\": {\"children\": [], \"name\": \"No issues\", \"id\": \"category4\"},\n",
84+
" \"MINOR_ISSUES\": {\"children\": [], \"name\": \"Minor issue(s)\", \"id\": \"category5\"},\n",
85+
" \"MAJOR_ISSUES\": {\"children\": [], \"name\": \"Major issue(s)\", \"id\": \"category6\"},\n",
86+
" },\n",
87+
" \"input\": \"radio\",\n",
88+
" },\n",
89+
" \"instruction\": \"Instructions Following\",\n",
90+
" \"level\": \"completion\",\n",
91+
" \"mlTask\": \"CLASSIFICATION\",\n",
92+
" \"required\": 0,\n",
93+
" \"isChild\": False,\n",
94+
" \"isNew\": False,\n",
95+
" },\n",
96+
" \"CLASSIFICATION_JOB_AT_COMPLETION_LEVEL_2\": {\n",
97+
" \"content\": {\n",
98+
" \"categories\": {\n",
99+
" \"NO_ISSUES\": {\"children\": [], \"name\": \"No issues\", \"id\": \"category7\"},\n",
100+
" \"MINOR_INACCURACY\": {\n",
101+
" \"children\": [],\n",
102+
" \"name\": \"Minor inaccuracy\",\n",
103+
" \"id\": \"category8\",\n",
104+
" },\n",
105+
" \"MAJOR_INACCURACY\": {\n",
106+
" \"children\": [],\n",
107+
" \"name\": \"Major inaccuracy\",\n",
108+
" \"id\": \"category9\",\n",
109+
" },\n",
110+
" },\n",
111+
" \"input\": \"radio\",\n",
112+
" },\n",
113+
" \"instruction\": \"Truthfulness\",\n",
114+
" \"level\": \"completion\",\n",
115+
" \"mlTask\": \"CLASSIFICATION\",\n",
116+
" \"required\": 0,\n",
117+
" \"isChild\": False,\n",
118+
" \"isNew\": False,\n",
119+
" },\n",
120+
" \"CLASSIFICATION_JOB_AT_COMPLETION_LEVEL_3\": {\n",
121+
" \"content\": {\n",
122+
" \"categories\": {\n",
123+
" \"NO_ISSUES\": {\"children\": [], \"name\": \"No issues\", \"id\": \"category10\"},\n",
124+
" \"MINOR_SAFETY_CONCERN\": {\n",
125+
" \"children\": [],\n",
126+
" \"name\": \"Minor safety concern\",\n",
127+
" \"id\": \"category11\",\n",
128+
" },\n",
129+
" \"MAJOR_SAFETY_CONCERN\": {\n",
130+
" \"children\": [],\n",
131+
" \"name\": \"Major safety concern\",\n",
132+
" \"id\": \"category12\",\n",
133+
" },\n",
134+
" },\n",
135+
" \"input\": \"radio\",\n",
136+
" },\n",
137+
" \"instruction\": \"Harmlessness/Safety\",\n",
138+
" \"level\": \"completion\",\n",
139+
" \"mlTask\": \"CLASSIFICATION\",\n",
140+
" \"required\": 0,\n",
141+
" \"isChild\": False,\n",
142+
" \"isNew\": False,\n",
143+
" },\n",
144+
" \"COMPARISON_JOB\": {\n",
145+
" \"content\": {\n",
146+
" \"options\": {\n",
147+
" \"IS_MUCH_BETTER\": {\"children\": [], \"name\": \"Is much better\", \"id\": \"option13\"},\n",
148+
" \"IS_BETTER\": {\"children\": [], \"name\": \"Is better\", \"id\": \"option14\"},\n",
149+
" \"IS_SLIGHTLY_BETTER\": {\n",
150+
" \"children\": [],\n",
151+
" \"name\": \"Is slightly better\",\n",
152+
" \"id\": \"option15\",\n",
153+
" },\n",
154+
" \"TIE\": {\"children\": [], \"name\": \"Tie\", \"mutual\": True, \"id\": \"option16\"},\n",
155+
" },\n",
156+
" \"input\": \"radio\",\n",
157+
" },\n",
158+
" \"instruction\": \"Pick the best answer\",\n",
159+
" \"mlTask\": \"COMPARISON\",\n",
160+
" \"required\": 1,\n",
161+
" \"isChild\": False,\n",
162+
" \"isNew\": False,\n",
163+
" },\n",
164+
" \"CLASSIFICATION_JOB_AT_ROUND_LEVEL\": {\n",
165+
" \"content\": {\n",
166+
" \"categories\": {\n",
167+
" \"BOTH_ARE_GOOD\": {\"children\": [], \"name\": \"Both are good\", \"id\": \"category17\"},\n",
168+
" \"BOTH_ARE_BAD\": {\"children\": [], \"name\": \"Both are bad\", \"id\": \"category18\"},\n",
169+
" },\n",
170+
" \"input\": \"radio\",\n",
171+
" },\n",
172+
" \"instruction\": \"Overall quality\",\n",
173+
" \"level\": \"round\",\n",
174+
" \"mlTask\": \"CLASSIFICATION\",\n",
175+
" \"required\": 0,\n",
176+
" \"isChild\": False,\n",
177+
" \"isNew\": False,\n",
178+
" },\n",
179+
" \"CLASSIFICATION_JOB_AT_CONVERSATION_LEVEL\": {\n",
180+
" \"content\": {\n",
181+
" \"categories\": {\n",
182+
" \"GLOBAL_GOOD\": {\"children\": [], \"name\": \"Globally good\", \"id\": \"category19\"},\n",
183+
" \"BOTH_ARE_BAD\": {\"children\": [], \"name\": \"Globally bad\", \"id\": \"category20\"},\n",
184+
" },\n",
185+
" \"input\": \"radio\",\n",
186+
" },\n",
187+
" \"instruction\": \"Global\",\n",
188+
" \"level\": \"conversation\",\n",
189+
" \"mlTask\": \"CLASSIFICATION\",\n",
190+
" \"required\": 0,\n",
191+
" \"isChild\": False,\n",
192+
" \"isNew\": False,\n",
193+
" },\n",
194+
" \"TRANSCRIPTION_JOB_AT_CONVERSATION_LEVEL\": {\n",
195+
" \"content\": {\"input\": \"textField\"},\n",
196+
" \"instruction\": \"Additional comments...\",\n",
197+
" \"level\": \"conversation\",\n",
198+
" \"mlTask\": \"TRANSCRIPTION\",\n",
199+
" \"required\": 0,\n",
200+
" \"isChild\": False,\n",
201+
" \"isNew\": False,\n",
202+
" },\n",
203+
" }\n",
204+
"}"
205+
]
206+
},
207+
{
208+
"cell_type": "markdown",
209+
"metadata": {},
210+
"source": [
211+
"Now, we create the project using the `create_project` method, with type `LLM_STATIC`:"
212+
]
213+
},
214+
{
215+
"cell_type": "code",
216+
"execution_count": null,
217+
"metadata": {},
218+
"outputs": [],
219+
"source": [
220+
"from kili.client import Kili\n",
221+
"\n",
222+
"kili = Kili(\n",
223+
" # api_endpoint=\"https://cloud.kili-technology.com/api/label/v2/graphql\",\n",
224+
")\n",
225+
"project = kili.create_project(\n",
226+
" title=\"[Kili SDK Notebook]: LLM Static\",\n",
227+
" description=\"Project Description\",\n",
228+
" input_type=\"LLM_STATIC\",\n",
229+
" json_interface=interface,\n",
230+
")\n",
231+
"project_id = project[\"id\"]"
232+
]
233+
},
234+
{
235+
"cell_type": "markdown",
236+
"metadata": {},
237+
"source": [
238+
"## Import conversations"
239+
]
240+
},
241+
{
242+
"cell_type": "markdown",
243+
"metadata": {},
244+
"source": [
245+
"We will import three conversations to the project. The conversations are stored in a JSON file, which we will load and import using the `import_conversations` method.\n"
246+
]
247+
},
248+
{
249+
"cell_type": "code",
250+
"execution_count": null,
251+
"metadata": {},
252+
"outputs": [],
253+
"source": [
254+
"import requests\n",
255+
"\n",
256+
"conversations = requests.get(\n",
257+
" \"https://storage.googleapis.com/label-public-staging/demo-projects/LLM_static/llm-conversations.json\"\n",
258+
").json()\n",
259+
"kili.llm.import_conversations(project_id, conversations=conversations)"
260+
]
261+
},
262+
{
263+
"cell_type": "markdown",
264+
"metadata": {},
265+
"source": [
266+
"You can now see the conversations imported in the UI :\n",
267+
"\n",
268+
"![Model Integration](./img/llm_conversations.png)"
269+
]
270+
},
271+
{
272+
"cell_type": "markdown",
273+
"metadata": {},
274+
"source": []
275+
},
276+
{
277+
"cell_type": "markdown",
278+
"metadata": {},
279+
"source": [
280+
"In this tutorial, we've:\n",
281+
"\n",
282+
"- **Created a Kili project** with a custom interface for LLM output comparison.\n",
283+
"- **Imported conversations** using Kili LLM format.\n"
284+
]
285+
}
286+
],
287+
"metadata": {
288+
"language_info": {
289+
"name": "python"
290+
}
291+
},
292+
"nbformat": 4,
293+
"nbformat_minor": 4
294+
}

‎src/kili/adapters/kili_api_gateway/asset/operations_mixin.py

+6-1
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,12 @@ def list_assets(
4242
project_info = get_project(
4343
self.graphql_client, filters.project_id, ("inputType", "jsonInterface")
4444
)
45-
if project_info["inputType"] in {"VIDEO", "LLM_RLHF", "LLM_INSTR_FOLLOWING"}:
45+
if project_info["inputType"] in {
46+
"VIDEO",
47+
"LLM_RLHF",
48+
"LLM_INSTR_FOLLOWING",
49+
"LLM_STATIC",
50+
}:
4651
yield from self.list_assets_split(filters, fields, options, project_info)
4752
return
4853

‎src/kili/adapters/kili_api_gateway/label/operations_mixin.py

+6-1
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,12 @@ def list_labels(
6161
project_info = get_project(
6262
self.graphql_client, filters.project_id, ("inputType", "jsonInterface")
6363
)
64-
if project_info["inputType"] in {"VIDEO", "LLM_RLHF", "LLM_INSTR_FOLLOWING"}:
64+
if project_info["inputType"] in {
65+
"VIDEO",
66+
"LLM_RLHF",
67+
"LLM_INSTR_FOLLOWING",
68+
"LLM_STATIC",
69+
}:
6570
yield from self.list_labels_split(filters, fields, options, project_info)
6671
return
6772

0 commit comments

Comments
 (0)