Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
137 lines (93 loc) · 6.32 KB

CONTRIBUTING.md

File metadata and controls

137 lines (93 loc) · 6.32 KB

OpenSearch dev environment

Running test OpenSearch instance

Prerequisites

  • NodeJS
  • Yarn (see package.json for version requirements for yarn and node)
  • Mage
node -v
yarn -v
mage -version

Build plugin

Run

yarn && yarn build && mage

to install the node_modules and build the OpenSearch plugin to the dist folder. This folder will be mounted in the Grafana container.

Run docker env

yarn server starts a single node OpenSearch cluster & OpenSearch Dashboards

If cluster not starting with error max virtual memory areas vm.max_map_count [65530] is too low..., increase mmap limits by running as root:

sysctl -w vm.max_map_count=262144

Add sample data

  1. Go to the kibana (http://localhost:5601)
  2. Login with admin:my_%New%_passW0rd!@#
  3. At the welcome screen click Add data and switch to the Sample data tab.
  4. Import Sample web logs and any other by your choice.

Data source configuration

URL: https://opensearch-node1:9200 Basic Auth: admin:my_%New%_passW0rd!@# Skip TLS Verify: true

How to set up local development environment to work with traces in Open Search

Open Search provides sample apps that are instrumented and can generate trace data out of the box. One of them is the Jaeger HotROD demo.

To set it up and add it as a Grafana datasource:

  1. Clone the Data Prepper project
  2. Go to the examples/jaeger-hotrod directory, then change the line image: jaegertracing/example-hotrod:latest to image: jaegertracing/example-hotrod:1.41.0 in docker-compose.yml
  3. Run docker compose up to start the containers
  • The app that generates the traces is at :8000. Clicking on the buttons in the app generates test traces.
  • The Opensearch Dashboards is at :5601 (Credentials are always user: admin, password: yourStrongPassword123!)
  • The Opensearch database is running at https://localhost:9200

In order to view generated traces in Grafana, add a new datasource, enter https://localhost:9200 as the url. Set “Skip TLS Verify” to true. The auth details are admin:yourStrongPassword123!. After following the steps for querying traces from ../README.md, traces from the sample app should be displayed in the panel.

E2E tests

  1. yarn playwright install --with-deps
  2. yarn server
  3. yarn e2e

Build a release

You need to have commit rights to the GitHub repository to publish a release.

  1. Update the version number in the package.json file.
  2. Update the CHANGELOG.md by copy and pasting the relevant PRs from Github's Release drafter interface or by running yarn generate-release-notes (you'll need to install the gh cli and jq to run this command).
  3. PR the changes.
  4. Once merged, follow the release process that you can find here

Client certificate authentication (TLS)

Prerequisite: Install openssl if necessary, for example brew install openssl on MacOS.

The repo opensearch-docker-compose brings together all the OpenSearch documentation on how to set up TLS client authentication in Docker, similar to the Basic Auth example above.

Using opensearch-docker-compose repo

  1. Clone the repo
  2. Follow the instructions to generate self-signed certificates
  3. Run docker-compose up as in their instructions
  4. Navigate to the OpenSearch Dashboard at https://localhost:5601/ and ingest sample data, for example Web Logs

...with Grafana

  1. Run Grafana locally with opensearch-datasource.
  2. On the configuration page:
    • URL: https://localhost:9200
    • TLS Client Auth: toggle on
    • Skip TLS Verify: toggle on
    • With CA Cert: toggle on
    • CA Cert: paste ca.pem (generated earlier)
    • Client Cert: admin.pem (generated earlier)
    • Client Key: admin.key (generated earlier)
    • Time field name: needs to correspond with data, for example if you added sample Web Logs, the name is timestamp

...with cURL

curl -XGET "https://localhost:9200/_msearch" -H 'Content-Type: application/json' --cert admin.pem --key admin.key --cacert ca.pem -d'
{ "index": "opensearch_dashboards_sample_data_logs"}
{ "query": { "match_all": {} }, "from": 0, "size": 10}
{ "index": "opensearch_dashboards_sample_data_ecommerce", "search_type": "dfs_query_then_fetch"}
{ "query": { "match_all": {} } }
'

Background

Similar to above, a cluster with OpenSearch and OpenSearch Dashboards is defined in a docker-compose. A script is used to generate certificates and keys for: Certificate Authority (ca), Admin (admin), and each node of the cluster (e.g. os01).

The OpenSearch docs explain it best: "Certificates are used to secure transport-layer traffic (node-to-node communication within your cluster) and REST-layer traffic (communication between a client and a node within your cluster)."

The Certificate Authority is usually an entity which manages these certificates on the web, but when developing locally we generate our own CA locally against which our keys are validated.

We generated a private key (client key) for the admin role in OpenSearch, then use the local CA we generated to create a public certificate (client certificate). This is the Client Certificate and Client Key which is entered into Grafana.

The keys, certificates, and CA are referenced in the configuration of OpenSearch (in the docker-compose.yml and in custom configuration opensearch.yml). The association between these certificates in OpenSearch and the certificates provided by the client ("admin" certificate in Grafana) is what facilitates the authentication.

References