Annotation queries - basic support. Updated readme, added todos

README.md

@@ -1,13 +1,81 @@
 # Wild GraphQL Datasource
 
-**This is work in progress and is not in a working state.**
+**This plugin is in early development. Breaking changes may happen at any time.**
 
 This is a Grafana datasource that aims to make requesting time series data via a GraphQL endpoint easy.
 This datasource is similar to https://github.com/fifemon/graphql-datasource, but is not compatible.
 This datasource tries to reimagine how GraphQL queries should be made from Grafana.
 
 Requests are made in the backend. Results are consistent between queries and alerting.
 
+
+## Variables
+
+### Provided Variables
+
+Certain variables are provided to every query. These include:
+
+| Variable      | Type   | Description                                                | Grafana counterpart                                                                                                |
+|---------------|--------|------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
+| `from`        | Number | Epoch milliseconds of the "from" time. Passed as a number. | [$__from](https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#__from-and-__to)    |
+| `to`          | Number | Epoch milliseconds of the "to" time. Passed as a number.   | [$__to](https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#__from-and-__to)      |
+| `interval_ms` | Number | Milliseconds of the interval. Equivalent to `to - from`.   | [$__interval_ms](https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/#__interval)  |
+
+An example usage is shown in the most basic query:
+
+```graphql
+query ($from: Long!, $to: Long!) {
+  queryStatus(from: $from, to: $to) {
+    # ...
+  }
+}
+```
+
+In the above example, the query asks for two Longs, `$from` and `$to`.
+The value is provided by the provided variables as seen in the above table.
+Notice that while `interval_ms` is provided, we do not use it or define it anywhere in our query.
+One thing to keep in mind for your own queries is the type accepted by the GraphQL server for a given variable.
+In the case of that specific schema, the type of a `Long` is allowed to be a number or a string.
+If your specific schema explicitly asks for a `String`, these variables may not work.
+Please [raise and issue](https://github.com/wildmountainfarms/wild-graphql-datasource/issues) if this limitation becomes a roadblock.
+
+
+### Query specific variables and Grafana variable interpolation
+
+For each query you define, the query editor has a place for you to add variables.
+This section is optional, but if you wish to provide constant variables for your query or [simplify the specification of input types](https://graphql.org/graphql-js/mutations-and-input-types/).
+You may do so.
+
+The variables section is the most useful for variable interpolation.
+Any value inside a string, whether that string is nested inside an object, or a top-most value of the variables object, can be interpolated.
+Please note that interpolation does not work for alerting queries.
+You may use any configuration of [variables](https://grafana.com/docs/grafana/latest/dashboards/variables/add-template-variables/) you see fit.
+An example is this:
+
+```graphql
+query ($sourceId: String!, $from: Long!, $to: Long!) {
+  queryStatus(sourceId: $sourceId, from: $from, to: $to) {
+    # ...
+  }
+}
+```
+
+```json
+{
+  "sourceId": "$sourceId"
+}
+```
+
+Here, `$sourceId` inside of the variables section will be interpolated with a value defined in your Grafana dashboard.
+`$sourceId` inside of the GraphQL query pane is a regular [variable](https://graphql.org/learn/queries/#variables) that is passed to the query.
+
+NOTE: Interpolating the entirety of the JSON text is not supported at this time.
+This means that interpolated variables cannot be passed as numbers and interpolated variables cannot define complex JSON objects.
+One of the pros of this is simplicity, with the advantage of not having to worry about escaping your strings.
+
+REMEMBER: Variable interpolation does not work for alerting queries, or any query that is executed without the frontend component.
+
+
 ## Uses for Timeseries data
 
 ### Using a field as the display name
@@ -23,6 +91,13 @@ References:
 * [partition by values](https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/)
   * Note: Partition by values was [added in 9.3](https://grafana.com/docs/grafana/latest/whatsnew/whats-new-in-v9-3/#new-transformation-partition-by-values) ([blog](https://grafana.com/blog/2022/11/29/grafana-9.3-release/))
 
+## FAQ
+
+* Can I use variable interpolation within the GraphQL query itself?
+  * No, but you may use variable interpolation inside the values of the variables passed to the query
+* Is this a drop-in replacement for [fifemon-graphql-datasource](https://grafana.com/grafana/plugins/fifemon-graphql-datasource/)?
+  * No, but both data sources have similar goals and can be ported between with little effort.
+
 ## Common errors
 
 ### Alerting Specific Errors
@@ -33,6 +108,15 @@ References:
 
 ## To-Do
 
+* Add ability to have multiple parsing options
+  * Add "advanced options" section that has "Partition by" and "Alias by"
+    * Including these might be necessary as you may want to partition by and alias by different things for different parts of a GraphQL query
+    * Advances options can also include the ability to add custom labels to the response - this allows different parsing options to be distinguished by Grafana
+* Add support for secure variable data defined in the data source configuration
+  * The variables defined here cannot be overridden for any request - this is for security
+  * Also add support for secure HTTP headers
+* See what minimum Grafana version we can support
+* Add support for variables: https://grafana.com/developers/plugin-tools/create-a-plugin/extend-a-plugin/add-support-for-variables
 * Add metrics to backend component: https://grafana.com/developers/plugin-tools/create-a-plugin/extend-a-plugin/add-logs-metrics-traces-for-backend-plugins#implement-metrics-in-your-plugin
 * Support returning logs data: https://grafana.com/developers/plugin-tools/tutorials/build-a-logs-data-source-plugin
   * We could just add `"logs": true` to `plugin.json`, however we need to support the renaming of fields because sometimes the `body` or `timestamp` fields will be nested

pkg/plugin/util/variables.go

@@ -7,3 +7,8 @@ func AutoPopulateVariables(query backend.DataQuery, variables *map[string]interf
 	(*variables)["to"] = query.TimeRange.To.UnixMilli()
 	(*variables)["interval_ms"] = query.Interval.Milliseconds()
 }
+
+// When we do get around to supporting variable substitution on the backend, we should make it as similar to the frontend as possible:
+//   https://github.com/grafana/grafana/blob/4b071f54529e24a2723eedf7ca4e7e989b3bd956/public/app/features/variables/utils.ts#L33
+//   The reason we would want variable substitution at all is for annotation queries because you cannot transform the result of those queries in any way.
+//   It might be possible to deal with that on the frontend, though.

src/components/QueryEditor.tsx

@@ -2,7 +2,7 @@ import React, {ChangeEvent, useEffect, useMemo} from 'react';
 import {InlineField, Input} from '@grafana/ui';
 import {CoreApp, QueryEditorProps} from '@grafana/data';
 import {DataSource} from '../datasource';
-import {getQueryVariablesAsJsonString, WildGraphQLDataSourceOptions, WildGraphQLMainQuery} from '../types';
+import {getQueryVariablesAsJsonString, WildGraphQLAnyQuery, WildGraphQLDataSourceOptions} from '../types';
 import {GraphiQLInterface} from 'graphiql';
 import {
   EditorContextProvider,
@@ -23,7 +23,7 @@ import {ExecutionResult} from "graphql-ws";
 import {AUTO_POPULATED_VARIABLES} from "../variables";
 
 
-type Props = QueryEditorProps<DataSource, WildGraphQLMainQuery, WildGraphQLDataSourceOptions>;
+type Props = QueryEditorProps<DataSource, WildGraphQLAnyQuery, WildGraphQLDataSourceOptions>;
 
 /**
  * This fetcher is designed to be used only for fetching the schema of a GraphQL endpoint.
@@ -55,6 +55,7 @@ function createFetcher(url: string, withCredentials: boolean, basicAuth?: string
   return async (graphQLParams: FetcherParams, opts?: FetcherOpts) => {
     const variables = {
       ...graphQLParams.variables, // TODO warn user if we are overriding their variables with the autopopulated ones
+      // TODO also consider if we want to override user variables
       ...AUTO_POPULATED_VARIABLES,
     };
     for (const field in variables) {

src/datasource.ts

@@ -1,21 +1,36 @@
-import {CoreApp, DataQueryRequest, DataQueryResponse, DataSourceInstanceSettings} from '@grafana/data';
+import {
+  AnnotationSupport,
+  CoreApp,
+  DataQueryRequest,
+  DataQueryResponse,
+  DataSourceInstanceSettings
+} from '@grafana/data';
 import {DataSourceWithBackend, getTemplateSrv} from '@grafana/runtime';
 import {Observable} from 'rxjs';
 
 import {
-  DEFAULT_ALERTING_QUERY,
+  DEFAULT_ALERTING_QUERY, DEFAULT_ANNOTATION_QUERY,
   DEFAULT_QUERY,
-  getQueryVariablesAsJson,
-  WildGraphQLDataSourceOptions,
-  WildGraphQLMainQuery
+  getQueryVariablesAsJson, WildGraphQLAnnotationQuery,
+  WildGraphQLAnyQuery,
+  WildGraphQLDataSourceOptions, WildGraphQLMainQuery
 } from './types';
 
-export class DataSource extends DataSourceWithBackend<WildGraphQLMainQuery, WildGraphQLDataSourceOptions> {
+export class DataSource extends DataSourceWithBackend<WildGraphQLAnyQuery, WildGraphQLDataSourceOptions> {
   settings: DataSourceInstanceSettings<WildGraphQLDataSourceOptions>;
+  annotations: AnnotationSupport<WildGraphQLAnnotationQuery>;
 
   constructor(instanceSettings: DataSourceInstanceSettings<WildGraphQLDataSourceOptions>) {
     super(instanceSettings);
     this.settings = instanceSettings;
+    this.annotations = {
+      // TODO annotation support is very minimal right now
+      // It works perfectly fine, however we need a way to add additional fields that can have template interpolation
+      //   to create informational labels for the annotation
+      getDefaultQuery(): Partial<WildGraphQLAnnotationQuery> {
+        return DEFAULT_ANNOTATION_QUERY;
+      }
+    };
   }
 
   getDefaultQuery(app: CoreApp): Partial<WildGraphQLMainQuery> {
@@ -26,11 +41,11 @@ export class DataSource extends DataSourceWithBackend<WildGraphQLMainQuery, Wild
     }
     return DEFAULT_QUERY;
   }
-  query(request: DataQueryRequest<WildGraphQLMainQuery>): Observable<DataQueryResponse> {
+  query(request: DataQueryRequest<WildGraphQLAnyQuery>): Observable<DataQueryResponse> {
 
     // Everything you see going on here is to do variable substitution for the values of the provided variables.
     const templateSrv = getTemplateSrv();
-    const newTargets: WildGraphQLMainQuery[] = request.targets.map((target) => {
+    const newTargets: WildGraphQLAnyQuery[] = request.targets.map((target) => {
       const variables = getQueryVariablesAsJson(target);
       const newVariables: any = { };
       for (const variableName in variables) {
@@ -49,4 +64,9 @@ export class DataSource extends DataSourceWithBackend<WildGraphQLMainQuery, Wild
     // we aren't really supposed to change this method, but we do it anyway :)
     return super.query(newRequest);
   }
+  // metricFindQuery(query: any, options?: any): Promise<MetricFindValue[]> {
+  //   // https://grafana.com/developers/plugin-tools/create-a-plugin/extend-a-plugin/add-support-for-variables
+  //   // Note that in the future, it looks like metricFindQuery will be deprecated in favor of variable support, similar in style to annotation support
+  //   return super.metricFindQuery(query, options);
+  // }
 }

src/plugin.json

@@ -6,6 +6,7 @@
   "metrics": true,
   "backend": true,
   "alerting": true,
+  "annotations": true,
   "executable": "gpx_wild_graphql_datasource",
   "info": {
     "description": "Grafana datasource to interpret GraphQL query results as timeseries data",

src/types.ts

@@ -6,7 +6,11 @@ type VariablesType = string | Record<string, any>;
 
 export interface ParsingOption {
   dataPath: string;
+  /** Required. The path to the time (this represents the "start time" in the case when {@link timeEndPath} is defined) */
   timePath: string;
+  // TODO use timeEndPath on the backend
+  /** Optional. The path to the "end time". Should only be shown for the annotation query. A blank string should be treated the same as undefined*/
+  timeEndPath?: string;
 }
 
 
@@ -55,6 +59,27 @@ export function getQueryVariablesAsJson(query: WildGraphQLCommonQuery): Record<s
 export interface WildGraphQLMainQuery extends WildGraphQLCommonQuery {
 }
 
+export interface WildGraphQLAnnotationQuery extends WildGraphQLCommonQuery {
+}
+
+/** This type represents the possible options that can be stored in the datasource JSON for queries */
+export type WildGraphQLAnyQuery = (WildGraphQLMainQuery | WildGraphQLAnnotationQuery) &
+  Partial<WildGraphQLMainQuery> &
+  Partial<WildGraphQLAnnotationQuery>;
+
+
+/**
+ * These are options configured for each DataSource instance
+ */
+export interface WildGraphQLDataSourceOptions extends DataSourceJsonData {
+}
+
+/**
+ * Value that is used in the backend, but never sent over HTTP to the frontend
+ */
+export interface WildGraphQLSecureJsonData {
+  // TODO We should support secret fields that can be passed to GraphQL queries as arguments
+}
 
 export const DEFAULT_QUERY: Partial<WildGraphQLMainQuery> = {
   queryText: `query BatteryVoltage($sourceId: String!, $from: Long!, $to: Long!) {
@@ -106,15 +131,22 @@ export const DEFAULT_ALERTING_QUERY: Partial<WildGraphQLMainQuery> = {
   ]
 };
 
-/**
- * These are options configured for each DataSource instance
- */
-export interface WildGraphQLDataSourceOptions extends DataSourceJsonData {
-}
-
-/**
- * Value that is used in the backend, but never sent over HTTP to the frontend
- */
-export interface WildGraphQLSecureJsonData {
-  // TODO We should support secret fields that can be passed to GraphQL queries as arguments
+export const DEFAULT_ANNOTATION_QUERY: Partial<WildGraphQLAnnotationQuery> = {
+  queryText: `query BatteryVoltage($from: Long!, $to: Long!) {
+  queryEvent(from:$from, to:$to) {
+    mateCommand {
+      dateMillis
+      packet {
+        commandName
+      }
+    }
+  }
 }
+`,
+  parsingOptions: [
+    {
+      dataPath: "queryEvent.mateCommand",
+      timePath: "dateMillis"
+    }
+  ]
+};

src/variables.ts

@@ -14,6 +14,7 @@ import {getTemplateSrv} from "@grafana/runtime";
  * Remember THE VALUES HERE ARE NOT USED BY THE BACKEND AND ARE ONLY USED FOR DEBUGGING QUERIES IN THE FRONTEND BY THE RUN BUTTON.
  */
 export const AUTO_POPULATED_VARIABLES: Record<string, any> = {
+  // TODO pass these values as numbers after interpolating them
   "from": "$__from",
   "to": "$__to",
   "interval_ms": "$__interval_ms",