From 8205548ef4f3034ebb3485c3524cfe5b1d50b4fc Mon Sep 17 00:00:00 2001 From: djl11 Date: Wed, 26 Feb 2025 12:57:44 +0700 Subject: [PATCH] moved all images and gifs inside Accordion elements, to prevent too much clutter. --- basics/welcome.mdx | 2 ++ interfaces/basics.mdx | 6 ++++++ interfaces/overview.mdx | 4 ++-- interfaces/plots.mdx | 18 ++++++++++++++++++ interfaces/tables.mdx | 34 ++++++++++++++++++++++++++++++++++ interfaces/views.mdx | 16 ++++++++++++++++ logging/column_contexts.mdx | 4 ++++ logging/contexts.mdx | 36 +++++++++++++++++++++++------------- logging/entries.mdx | 4 ++++ logging/experiments.mdx | 6 ++++++ logging/logs.mdx | 10 ++++++++++ logging/overview.mdx | 4 ++-- logging/parameters.mdx | 8 ++++++++ logging/traces.mdx | 4 ++++ universal_api/overview.mdx | 8 ++++---- 15 files changed, 143 insertions(+), 21 deletions(-) diff --git a/basics/welcome.mdx b/basics/welcome.mdx index 60a02fc4e..6b53c465b 100644 --- a/basics/welcome.mdx +++ b/basics/welcome.mdx @@ -2,6 +2,8 @@ title: 'Welcome to Unify!' --- + + Unify is a **fully hackable** LLMOps platform, which you can use to build *personalized* pipelines for: logging, evaluations, guardrails, human labelling, agentic workflows, self-optimization, and more. simply `unify.log` your data, and then compose your own custom interface using the four core building blocks: (1) **tables**, (2) **plots**, (3) **visualizations**, and (4) **terminals**. diff --git a/interfaces/basics.mdx b/interfaces/basics.mdx index b3f7930ba..5e826ba5b 100644 --- a/interfaces/basics.mdx +++ b/interfaces/basics.mdx @@ -30,7 +30,9 @@ Tabs consist of **tiles**, which is where the data, interaction, and visualizati When in edit mode (`Ctrl + E`), new tiles can be added by clicking on any blank space in the tab, pressing `Ctrl + T`, or clicking "+ Add Tile" at the top of the screen. The corners of the tab can then be dragged for resizing, and the tab can be dragged and repositioned. + + If you're done positioning and sizing your tiles, it's best to exit edit mode. This will declutter the workspace, minimize the icons on screen and make the best use of all the tile space. @@ -42,7 +44,9 @@ by clicking the expand button in the top right of the tile. This will open the tile in focus mode, where other tiles can be positioned alongside for comparison. You can exit via the top right "X" button to return to the default interface view. + + Focus mode can also be accessed from the focus mode button at the top the default interface screen. @@ -52,7 +56,9 @@ Edit mode enables tile manipulation (as explained above) and interactive mode en such as applying filters, sorting and grouping on tables, changing the axes selection on plots, and editing logs via the view tile. When interactive mode is turned off, the interface acts more like a static dashboard, with data manipulation features disabled. + + ## Contexts diff --git a/interfaces/overview.mdx b/interfaces/overview.mdx index a03ebdf3d..7388bf41f 100644 --- a/interfaces/overview.mdx +++ b/interfaces/overview.mdx @@ -4,6 +4,8 @@ title: 'Overview' Build custom interfaces for: logging, evals, guardrails, labelling, tracing, agents, human-in-the-loop, hyperparam sweeps, and anything else you can think of ✨ + + Just `unify.log` your data, and add an interface using the four building blocks: - tables 🔢 @@ -13,8 +15,6 @@ Just `unify.log` your data, and add an interface using the four building blocks: Every LLM product has unique and changing requirements, as do the users. Your infra should reflect this! - - We've tried to make Unify as **(a) simple**, **(b) modular** and **(c) hackable** as possible, so you can quickly probe, analyze, and iterate on the data that's important for **you**, your **product** and your **users** ⚡ diff --git a/interfaces/plots.mdx b/interfaces/plots.mdx index be0fa55c4..a6da334a8 100644 --- a/interfaces/plots.mdx +++ b/interfaces/plots.mdx @@ -26,7 +26,9 @@ for i in range(10): We can then select the x column for the x-axis and the y column for the y-axis. + + We can also add overlay several plots on the graph, by selecting another column to group by (based on value equality, the same as grouping in the table). @@ -50,7 +52,9 @@ for gender in ["male", "female"]: We can then select age for the x-axis, salary for the y-axis and group by gender. We can also add a line of best fit to each group in the plot. + + If we want to group across *multiple* independent variables, we can just create a new derived column to express the desired group. Let's assume we also log the nationality of the person. @@ -76,12 +80,16 @@ for gender in ["male", "female"]: In order to plot against both variables, let's create a new derived column called `gender + nationality` with the equation `gender + nationality`. We can then group by this new column. + + We could then create three plots for example. One which groups by gender, one which groups by location and one which groups by both. We can then save this and revisit this dashboard again any time in future, whenever more data is added into the project. + + ## Line Graphs @@ -97,7 +105,9 @@ for i in range(10): ) ``` + + As before, we can also group by another column. Let's imagine we want to plot the speed of different models across time, @@ -120,7 +130,9 @@ while True: We can then select time for the x-axis, speed for the y-axis and group by model. + + By default, new data will **not** be loaded into the interface unless something triggers a reload, such as filtering, grouping or sorting. @@ -129,7 +141,9 @@ in order to continually stream new data into the platform. if we want a moving window of the runtime, we can turn streaming on for both the table and the plot, and then limit the time horizon to the last 5 minutes, for example. + + Similar to the derived "identity" column before, we can create a derived "endpoint" column to plot a unique line for each model + provider combination. @@ -163,7 +177,9 @@ for i in range(25): Let's then create the derived column and plot results for each unique **endpoint**. + + ## Bar Charts @@ -223,6 +239,8 @@ for month in range(1, 13): Let's take a look at the corresponding histogram. + + We can see that the usage has been steadily increasing throughout the year, good stuff! 📈 diff --git a/interfaces/tables.mdx b/interfaces/tables.mdx index 0f265155b..5d4952185 100644 --- a/interfaces/tables.mdx +++ b/interfaces/tables.mdx @@ -14,7 +14,9 @@ unify.log(x=1, y={"b": [4, 5, 6]}, msg="goodbye", score=0.456) This will be shown as: + + Note that the **most recent** log is always shown at the **top** of the table, with row numbers in an ascending manner from top to bottom. @@ -29,20 +31,26 @@ Columns can easily be hidden by clicking the dropdown and pressing "hide column" - hovering between columns and pressing the "+" button - clicking the column visibility dropdown and toggling the column + + ### Moving Columns You can move columns by simply dragging them to where you want them. + + ### Pinning Columns You can pin columns to the left by dragging the left-most edge of the column towards the right (think Excel). This column will then be locked in place when you scroll left and right within the table. + + ## Grouping @@ -81,17 +89,23 @@ By default, the data is just shown flat, in the order it was logged (most recent However, if we want to see the results of each unique experiment, we can simply group by the experiment column, and unfold the data we want to see. The reduction metrics are automatically calculated for each column. + + Conversely, if we want to see which question was the most difficult, we can group by question instead. + + We can see that the LLM seems to find London the easiest, then Tokyo, then Paris (again, a contrived example 😁). Similarly, if we want to what the impact of a system message is, we can group by this column instead, and see the statics reduced across all experiments. + + Combined, this makes it very easily to quickly probe the data across **all** of your experiments, enabling you to easily transcending the boundaries of rigid "experiments" flexibly. @@ -100,7 +114,9 @@ Combined, this makes it very easily to quickly probe the data across **all** of As you'd expect, sorting can also be applied to any column. Simply clicking "sort" will toggle the state between ascending, descending, and off. Rows are sorted in the order of the column sorts. For example, sorting by column A and *then* column B will prioritize sorting based on A, and only defer to B when the values in Column A are equal. + + ### With Grouping [🚧] @@ -185,13 +201,17 @@ Everything else is column specific. - `Includes`: whether or not the string includes the substring - `Excludes`: whether or not the string excludes the substring + + ### Numbers - `==`, `!=`, `>`, `>=`, `<`, `<=`: standard comparison operators + + ### DateTime @@ -199,21 +219,27 @@ Everything else is column specific. - `Relative Mode`: select relative to the current time - `Absolute Mode`: select an absolute point in time + + ### Booleans - `True`: is true - `False`: is false + + ### Lists and Dicts - `Includes`: whether or not the list includes the item / dict includes the key - `Excludes`: whether or not the list excludes the item / dict excludes the key + + ### Groups [🚧] @@ -238,12 +264,16 @@ unify.log(x=1, y=1) Let's imagine we want to filter to find the logs where `x * y > 0`. We couldn't achieve this with column filters, but we can with the global filter, using an expression such as `x * y > 0` or `( x > 0 and y > 0) or (x < 0 and y < 0)`. + + We can also switch the global filter to "search" mode, which simply searches all cells for a matching substring, after casting all contents to string form (including dicts, lists, etc). + + ## Derived Columns @@ -259,12 +289,16 @@ for _ in range(20): We just need to provide the expression `(x**2 + y**2)**0.5`: + + Filtering, grouping, and sorting all work on derived columns, just like any other. However, derived columns *also* support updating their equation. + + ## Joins [🚧] diff --git a/interfaces/views.mdx b/interfaces/views.mdx index 349ee6b73..efc1df646 100644 --- a/interfaces/views.mdx +++ b/interfaces/views.mdx @@ -6,7 +6,9 @@ Views enable us to actually **look** at the data, not only filter, sort, organiz A view tile must be paired with exactly one table, this can be done from the menu at the top. + + ## Cell Selection @@ -23,7 +25,9 @@ GIF To avoid redundancy, every value that is shared across multiple cells is grouped in the view pane, with all the row numbers shown at the top. + + This also makes it easy to quickly spot trends in the data. @@ -35,7 +39,9 @@ The view pane supports 3 different view modes: 2. **Plain Text**: All text is rendered as plain text. 3. **Raw**: The columns (incuding dicts, lists, numbers etc.) are rendered as a raw strings, with no formatting or nested structure. + + ## Hiding Columns @@ -69,13 +75,17 @@ for question in ["what is 1 + 1?", "what is 2 + 2?", "what is 3 + 3?"]: The most intuitive way to view this data is to show the numeric data in the table (with all the power of sorting, grouping, filtering etc.), and the text data in the view pane to look at the finer details (and benefit from all the features explained below!) + + ## Moving Columns Columns can easily be moved in the view pane by selecting drag mode and then simply clicking and dragging them. + + ## Split View @@ -136,19 +146,25 @@ For string diffs, there are four modes: - **Side-by-side**: Per word diffs. - **Characters**: Per character diffs. + + ### Numeric Numeric value "diffs" support the four basic operators: `-`, `+`, `*` and `/`. + + ### DateTime Datetime diffs are presented as a relative difference in time, in the appropriate units (weeks, days, seconds, miliseconds etc.) + + ### Structure Diffs diff --git a/logging/column_contexts.mdx b/logging/column_contexts.mdx index 8c5aff8c4..d28587f90 100644 --- a/logging/column_contexts.mdx +++ b/logging/column_contexts.mdx @@ -21,7 +21,9 @@ with unify.Log(), unify.ColumnContext("Sciences"): ) ``` + + Under the hood, column contexts are simply `/` separated field names, and so the below code is equivalent. @@ -46,7 +48,9 @@ with unify.Log(): Columns contexts can be **moved**, **hidden**, and **focused on** in the table, making it easy to view and organize your structured data in bulk. + + As with contexts, `unify.ColumnContext("...")` by default will also change the behaviour of `get_logs`, returning only the logs which are part of the column context. diff --git a/logging/contexts.mdx b/logging/contexts.mdx index 9a5b01814..0e058c07c 100644 --- a/logging/contexts.mdx +++ b/logging/contexts.mdx @@ -12,7 +12,8 @@ When no context is specified in the interface or the table, then the default con However, you can group your logs into *non-overlapping* contexts, which can then be displayed in separate tables. ```python -names = ["Zoe", "John", "Jane", "Jim", "Jill"] +first_names = ["Zoe", "John", "Jane", "Jim", "Jill"] +last_names = ["Smith", "Johnson", "Williams", "Jones", "Brown"] qnas = { "what is 1 + 1?": 2, "what is 2 * 3?": 6, @@ -23,16 +24,22 @@ qnas = { regions = ["US", "EU", "UK", "AU", "CA"] for _ in range(20): - question, answer = random.choice( - list(qnas.items()) + question = random.choice( + list(qnas.keys()) ) unify.log( - name=random.choice(names), + timestamp=datetime.now().isoformat(), + first_name=random.choice(first_names), + last_name=random.choice(last_names), question=question, - answer=answer, region=random.choice(regions), + dob=datetime( + year=random.randint(1980, 2010), + month=random.randint(1, 12), + day=random.randint(1, 28) + ).isoformat(), context="Traffic" -) + ) predictions = [ "six", "one", "two", "three", "four", "five" @@ -43,9 +50,8 @@ for _ in range(15): list(qnas.items()) ) unify.log( - name=random.choice(names), question=question, - prediction=random.choice( + agent_prediction=random.choice( predictions ), answer=answer, @@ -53,8 +59,9 @@ for _ in range(15): ) ``` -{/* ToDo: Add context improvements (https://app.clickup.com/t/86c27zpkq) */} -{/* GIF */} + + + ## Context Handling @@ -74,8 +81,6 @@ with unify.Context("Traffic"): ) ``` -IMG - These can also be arbitrarily nested, where each context string is joined via `/` to define the full context path. ```python @@ -94,7 +99,12 @@ with unify.Context("Sciences"): ) ``` -IMG +The "Sciences" context cannot itself be selected in a table (there is no data), +but it forms part of the directory structure for selecting the available contexts. +However, "Sciences" **can** be selected as the context of the tab in the interface, +which limits the search space for each table within the tab. + +GIF By default, `unify.Context("...")` also changes the behaviour of `get_logs`, returning only the logs relevant within the context. diff --git a/logging/entries.mdx b/logging/entries.mdx index 5adb61308..71c07f9eb 100644 --- a/logging/entries.mdx +++ b/logging/entries.mdx @@ -43,7 +43,9 @@ for grade, students in student_data.items(): Which will appear like so in the table: + + These can also be nested: @@ -93,7 +95,9 @@ for subject, studate_data in all_student_data.items(): Which will appear in the table like so: + + Of course, this is not especially useful for the above example, but if there are many nested function calls processing the data inside the `subject` or `grade` contexts, diff --git a/logging/experiments.mdx b/logging/experiments.mdx index 79686a3fe..91d9f40c4 100644 --- a/logging/experiments.mdx +++ b/logging/experiments.mdx @@ -20,7 +20,9 @@ with unify.Experiment(), unify.Params( This will show in the table like so: + + Let's run another experiment, but give it an explicit name. @@ -37,7 +39,9 @@ with unify.Experiment("my_experiment"), unify.Params( This will show in the table like so: + + You can also use negative indexing, and the `overwrite` argument. For example, the following will overwrite the previous experiment. @@ -57,4 +61,6 @@ with unify.Experiment(-1, overwrite=True), unify.Params( The overwrite can be seen in the table, after pressing the `refresh` button: + + diff --git a/logging/logs.mdx b/logging/logs.mdx index 652e3ea87..4217cb9bc 100644 --- a/logging/logs.mdx +++ b/logging/logs.mdx @@ -10,7 +10,9 @@ unify.log(x=0, y={"a": [1, 2, 3]}, msg="hello", score=0.123) Logs appear as **rows** in tables, and fields appear as **columns** + + ## Logging Contexts @@ -24,7 +26,9 @@ with unify.Log(): unify.log(z=2) ``` + + These can also be arbitrarily nested. Note that the most recently **created** log is always shown at the top of the table (not most recently **updated**). @@ -40,7 +44,9 @@ with unify.Log(): unify.log(z=2) ``` + + We can also pass arguments, which will also be part of the same log. @@ -49,7 +55,9 @@ with unify.Log(x=0, y=1): unify.log(z=2) ``` + + Using `with unify.Log()` is the best practice, as this avoids the need to pass explicit log handles throughout your program. This code uses explicit handles (each function has a `log` argument): @@ -90,4 +98,6 @@ with unify.Log(a=a): Both result in the same log: + + diff --git a/logging/overview.mdx b/logging/overview.mdx index 16d89a639..9c448985b 100644 --- a/logging/overview.mdx +++ b/logging/overview.mdx @@ -4,6 +4,8 @@ title: 'Overview' Logs enable you to store **any** kind of data to use in your [interfaces](https://console.unify.ai/) 📊 + + All logging features are supported the [REST API](https://docs.unify.ai/api-reference), but our Python client provides more convenient abstractions on top of the basic logging primitives. @@ -16,8 +18,6 @@ unify.log(x=0, y={"a": [1, 2, 3]}, msg="hello", score=0.123) ...and you're ready to go 🚀 - - We've tried to make Unify as **(a) simple**, **(b) modular** and **(c) hackable** as possible, so you can quickly probe, analyze, and iterate on the data that's important for **you**, your **product** and your **users** ⚡ diff --git a/logging/parameters.mdx b/logging/parameters.mdx index 40c1098fa..b24f965f2 100644 --- a/logging/parameters.mdx +++ b/logging/parameters.mdx @@ -23,14 +23,18 @@ log = unify.log( They are always shown on the left hand side of the table. + + Parameter versions are of type string, and the version is simply an incrementing integer. In future, we plan to support custom version naming, thus the choice of a *string* type rather than an *integer*. You can see the version of a paramter in the view pane. + + ## Parameter Contexts @@ -48,7 +52,9 @@ with unify.Params( This will show in the table like so: + + Like the other context managers, parameter context managers can also be nested: @@ -68,7 +74,9 @@ with unify.Params( This will show in the table like so: + + As with entries, `unify.Parameter("...")` also changes the behaviour of `get_logs`, returning only the logs which include the parameter. This behaviour can be change by setting `unify.Parameter("...", mode="read")` which will only impact log **getting**, diff --git a/logging/traces.mdx b/logging/traces.mdx index d6348deb3..5768d2de8 100644 --- a/logging/traces.mdx +++ b/logging/traces.mdx @@ -14,7 +14,9 @@ my_function(1, 2) If you create a view tile and select the trace in the table, you'll then see the trace viewer. + + Traces are made up of spans, each of which represent a unit of computation. Traces can have an arbitrary number of spans, and spans can have child spans up to an arbitrary depth. @@ -40,7 +42,9 @@ outer_fn(3, 4) In the trace view pane, this will look as follows: + + ## LLM Traces diff --git a/universal_api/overview.mdx b/universal_api/overview.mdx index c1aac414a..ad9959db5 100644 --- a/universal_api/overview.mdx +++ b/universal_api/overview.mdx @@ -2,8 +2,10 @@ title: 'Overview' --- -New LLM models come out all the time, with each provider requiring a unique API key, billing, and API standard. -It quickly becomes cumbersome trying to manage all of these nuances. +New LLM models come out all the time, with each provider requiring unique API keys, billing, and formatting. +We've unified all of these into a single API, so you can focus on what matters 🎯 + + Our Universal API provides: @@ -11,8 +13,6 @@ Our Universal API provides: - One account, with one balance and one API key 🔑 - Integrations for your own fine-tuned and/or local LLMs 🖥️ - - To get started, just [sign up](https://console.unify.ai/), `pip install unifyai`, and make your first query: