Skip to content

Commit d3c069b

Browse files
walterraFil
walterra
and
Fil
authored
Data loader elasticsearch (#1490)
* Add Elasticsearch example * Fix links * adds info how to set up elasticsearch/kibana * fix README.md * fix instructions to include NODE_CA_FINGERPRINT * adds support for NODE_CA_FINGERPRINT to es_client.ts * fix comment about ES_UNSAFE_TLS_REJECT_UNAUTHORIZED * tweak loader * remove postgres title * Update examples/loader-elasticsearch/src/index.md tweak text * Update examples/loader-elasticsearch/src/index.md adding empty lines should allow framework to convert the markdown (in this case, the backticks around .env) * Update examples/loader-elasticsearch/src/index.md adding empty lines should allow framework to convert the markdown (in this case, the backticks around .env) * add link to examples/README.md * PR feedback: update obshq; mention dataset * Update examples/loader-elasticsearch/README.md fix quote * Update examples/loader-elasticsearch/README.md fix quotes --------- Co-authored-by: Philippe Rivière <[email protected]>
1 parent d76e803 commit d3c069b

File tree

10 files changed

+570
-0
lines changed

10 files changed

+570
-0
lines changed

‎examples/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
- [`loader-arrow`](https://observablehq.observablehq.cloud/framework-example-loader-arrow/) - Generating Apache Arrow IPC files
5757
- [`loader-databricks`](https://observablehq.observablehq.cloud/framework-example-loader-databricks/) - Loading data from Databricks
5858
- [`loader-duckdb`](https://observablehq.observablehq.cloud/framework-example-loader-duckdb/) - Processing data with DuckDB
59+
- [`loader-elasticsearch`](https://observablehq.observablehq.cloud/framework-example-loader-elasticsearch/) - Loading data from Elasticsearch
5960
- [`loader-github`](https://observablehq.observablehq.cloud/framework-example-loader-github/) - Loading data from GitHub
6061
- [`loader-google-analytics`](https://observablehq.observablehq.cloud/framework-example-loader-google-analytics/) - Loading data from Google Analytics
6162
- [`loader-julia-to-txt`](https://observablehq.observablehq.cloud/framework-example-loader-julia-to-txt/) - Generating TXT from Julia

‎examples/loader-elasticsearch/.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.DS_Store
2+
.env
3+
/dist/
4+
node_modules/
5+
yarn-error.log

‎examples/loader-elasticsearch/README.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
[Framework examples →](../)
2+
3+
# Elasticsearch data loader
4+
5+
View live: <https://observablehq.observablehq.cloud/framework-example-loader-elasticsearch/>
6+
7+
This Observable Framework example demonstrates how to write a TypeScript data loader that runs a query on Elasticsearch using the [Elasticsearch Node.js client](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html). The data loader lives in [`src/data/kibana_sample_data_logs.csv.ts`](./src/data/kibana_sample_data_logs.csv.ts) and uses the helper [`src/data/es_client.ts`](./src/data/es_client.ts).
8+
9+
To fully reproduce the example, you need to have a setup with both Elasticsearch and Kibana running to create the sample data. Here’s how to set up both on macOS:
10+
11+
```bash
12+
# Download and run Elasticsearch
13+
curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.1-darwin-x86_64.tar.gz
14+
gunzip -c elasticsearch-8.14.1-darwin-x86_64.tar.gz | tar xopf -
15+
cd elasticsearch-8.14.1
16+
./bin/elasticsearch
17+
18+
# Next, in another terminal tab, download and run Kibana
19+
curl -O https://artifacts.elastic.co/downloads/kibana/kibana-8.14.1-darwin-x86_64.tar.gz
20+
gunzip -c kibana-8.14.1-darwin-x86_64.tar.gz | tar xopf -
21+
cd kibana-8.14.1
22+
./bin/kibana
23+
```
24+
25+
The commands for both will output instructions how to finish the setup with security enabled. Once you have both running, you can create the “Sample web logs” dataset in Kibana via this URL: http://localhost:5601/app/home#/tutorial_directory/sampleData
26+
27+
Finally, create the `.env` file with the credentials shared for the user `elastic` that were logged when starting Elasticsearch like this. To get the CA fingerprint for the config, run the following command from the directory you started installing Elasticsearch:
28+
29+
```
30+
openssl x509 -fingerprint -sha256 -noout -in ./elasticsearch-8.14.1/config/certs/http_ca.crt
31+
```
32+
33+
```
34+
ES_NODE="https://elastic:<PASSWORD>@localhost:9200"
35+
ES_CA_FINGERPRINT="<CA_FINGERPRINT>"
36+
ES_UNSAFE_TLS_REJECT_UNAUTHORIZED="FALSE"
37+
```

‎examples/loader-elasticsearch/observablehq.config.js

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
export default {
2+
root: "src"
3+
};

‎examples/loader-elasticsearch/package.json

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"type": "module",
3+
"private": true,
4+
"scripts": {
5+
"clean": "rimraf src/.observablehq/cache",
6+
"build": "rimraf dist && observable build",
7+
"dev": "observable preview",
8+
"deploy": "observable deploy",
9+
"observable": "observable"
10+
},
11+
"dependencies": {
12+
"@elastic/elasticsearch": "^8.14.0",
13+
"@observablehq/framework": "latest",
14+
"d3-dsv": "^3.0.1",
15+
"d3-time": "^3.1.0",
16+
"dotenv": "^16.4.5"
17+
},
18+
"devDependencies": {
19+
"@types/d3-dsv": "^3.0.7",
20+
"@types/d3-time": "^3.0.3",
21+
"rimraf": "^5.0.5"
22+
},
23+
"engines": {
24+
"node": ">=20"
25+
}
26+
}

‎examples/loader-elasticsearch/src/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/.observablehq/cache/

‎examples/loader-elasticsearch/src/data/es_client.ts

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
import "dotenv/config";
2+
import { Client } from "@elastic/elasticsearch";
3+
4+
// Have a look at the "Getting started" guide of the Elasticsearch node.js client
5+
// to learn more about how to configure these environment variables:
6+
// https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/getting-started-js.html
7+
8+
const {
9+
// ES_NODE can include the username and password in the URL, e.g.:
10+
// ES_NODE=https://<USERNAME>:<PASSWORD>@<HOST>:9200
11+
ES_NODE,
12+
// As an alternative to ES_NODE when using Elastic Cloud, you can use ES_CLOUD_ID and
13+
// set it to the Cloud ID that you can find in the cloud console of the deployment (https://cloud.elastic.co/).
14+
ES_CLOUD_ID,
15+
// ES_API_KEY can be used instead of username and password.
16+
// The API key will take precedence if both are set.
17+
ES_API_KEY,
18+
ES_USERNAME,
19+
ES_PASSWORD,
20+
// the fingerprint (SHA256) of the CA certificate that is used to sign
21+
// the certificate that the Elasticsearch node presents for TLS.
22+
ES_CA_FINGERPRINT,
23+
// Warning: This option should be considered an insecure workaround for local development only.
24+
// You may wish to specify a self-signed certificate rather than disabling certificate verification.
25+
// ES_UNSAFE_TLS_REJECT_UNAUTHORIZED can be set to FALSE to disable certificate verification.
26+
// See https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/client-connecting.html#auth-tls for more.
27+
ES_UNSAFE_TLS_REJECT_UNAUTHORIZED,
28+
} = process.env;
29+
30+
if ((!ES_NODE && !ES_CLOUD_ID) || (ES_NODE && ES_CLOUD_ID))
31+
throw new Error(
32+
"Either ES_NODE or ES_CLOUD_ID need to be defined, but not both.",
33+
);
34+
35+
const esUrl = ES_NODE ? new URL(ES_NODE) : undefined;
36+
const isHTTPS = esUrl?.protocol === "https:";
37+
const isLocalhost = esUrl?.hostname === "localhost";
38+
39+
export const esClient = new Client({
40+
...(ES_NODE ? { node: ES_NODE } : {}),
41+
...(ES_CLOUD_ID ? { cloud: { id: ES_CLOUD_ID } } : {}),
42+
...(ES_CA_FINGERPRINT ? { caFingerprint: ES_CA_FINGERPRINT } : {}),
43+
...(ES_API_KEY
44+
? {
45+
auth: {
46+
apiKey: ES_API_KEY,
47+
},
48+
}
49+
: {}),
50+
...(!ES_API_KEY && ES_USERNAME && ES_PASSWORD
51+
? {
52+
auth: {
53+
username: ES_USERNAME,
54+
password: ES_PASSWORD,
55+
},
56+
}
57+
: {}),
58+
...(isHTTPS &&
59+
isLocalhost &&
60+
ES_UNSAFE_TLS_REJECT_UNAUTHORIZED?.toLowerCase() === "false"
61+
? {
62+
tls: {
63+
rejectUnauthorized: false,
64+
},
65+
}
66+
: {}),
67+
});

‎examples/loader-elasticsearch/src/data/kibana_sample_data_logs.csv

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
date,count,response_code
2+
2024-06-16,230,200
3+
2024-06-16,12,503
4+
2024-06-16,7,404
5+
2024-06-17,209,200
6+
2024-06-17,12,404
7+
2024-06-17,10,503
8+
2024-06-18,210,200
9+
2024-06-18,16,404
10+
2024-06-18,4,503
11+
2024-06-19,220,200
12+
2024-06-19,11,404
13+
2024-06-19,5,503
14+
2024-06-20,211,200
15+
2024-06-20,15,404
16+
2024-06-20,4,503
17+
2024-06-21,210,200
18+
2024-06-21,12,404
19+
2024-06-21,8,503
20+
2024-06-22,211,200
21+
2024-06-22,10,503
22+
2024-06-22,8,404
23+
2024-06-23,219,200
24+
2024-06-23,9,404
25+
2024-06-23,3,503
26+
2024-06-24,211,200
27+
2024-06-24,12,404
28+
2024-06-24,7,503
29+
2024-06-25,208,200
30+
2024-06-25,11,404
31+
2024-06-25,11,503
32+
2024-06-26,222,200
33+
2024-06-26,7,503
34+
2024-06-26,1,404
35+
2024-06-27,209,200
36+
2024-06-27,13,503
37+
2024-06-27,8,404
38+
2024-06-28,203,200
39+
2024-06-28,18,404
40+
2024-06-28,9,503
41+
2024-06-29,216,200
42+
2024-06-29,8,404
43+
2024-06-29,6,503
44+
2024-06-30,214,200
45+
2024-06-30,10,404
46+
2024-06-30,6,503
47+
2024-07-01,212,200
48+
2024-07-01,12,404
49+
2024-07-01,6,503
50+
2024-07-02,215,200
51+
2024-07-02,12,404
52+
2024-07-02,3,503
53+
2024-07-03,212,200
54+
2024-07-03,11,404
55+
2024-07-03,6,503
56+
2024-07-04,210,200
57+
2024-07-04,14,404
58+
2024-07-04,7,503
59+
2024-07-05,211,200
60+
2024-07-05,14,404
61+
2024-07-05,5,503
62+
2024-07-06,196,200
63+
2024-07-06,32,404
64+
2024-07-06,2,503
65+
2024-07-07,205,200
66+
2024-07-07,17,404
67+
2024-07-07,8,503
68+
2024-07-08,204,200
69+
2024-07-08,14,404
70+
2024-07-08,12,503
71+
2024-07-09,202,200
72+
2024-07-09,17,404
73+
2024-07-09,11,503
74+
2024-07-10,214,200
75+
2024-07-10,10,503
76+
2024-07-10,6,404
77+
2024-07-11,206,200
78+
2024-07-11,13,404
79+
2024-07-11,11,503
80+
2024-07-12,218,200
81+
2024-07-12,6,404
82+
2024-07-12,6,503
83+
2024-07-13,216,200
84+
2024-07-13,10,404
85+
2024-07-13,4,503
86+
2024-07-14,216,200
87+
2024-07-14,7,404
88+
2024-07-14,6,503
89+
2024-07-15,218,200
90+
2024-07-15,12,404
91+
2024-07-15,1,503
92+
2024-07-16,162,200
93+
2024-07-16,7,404
94+
2024-07-16,4,503
95+
2024-07-17,211,200
96+
2024-07-17,14,404
97+
2024-07-17,5,503
98+
2024-07-18,214,200
99+
2024-07-18,9,404
100+
2024-07-18,7,503
101+
2024-07-19,213,200
102+
2024-07-19,9,404
103+
2024-07-19,8,503
104+
2024-07-20,212,200
105+
2024-07-20,10,404
106+
2024-07-20,7,503
107+
2024-07-21,203,200
108+
2024-07-21,21,404
109+
2024-07-21,7,503
110+
2024-07-22,215,200
111+
2024-07-22,8,503
112+
2024-07-22,7,404
113+
2024-07-23,208,200
114+
2024-07-23,17,404
115+
2024-07-23,4,503
116+
2024-07-24,212,200
117+
2024-07-24,13,404
118+
2024-07-24,6,503
119+
2024-07-25,211,200
120+
2024-07-25,13,404
121+
2024-07-25,6,503
122+
2024-07-26,213,200
123+
2024-07-26,9,404
124+
2024-07-26,8,503
125+
2024-07-27,210,200
126+
2024-07-27,110,404
127+
2024-07-27,9,503
128+
2024-07-28,217,200
129+
2024-07-28,8,404
130+
2024-07-28,6,503
131+
2024-07-29,200,200
132+
2024-07-29,17,404
133+
2024-07-29,13,503
134+
2024-07-30,215,200
135+
2024-07-30,12,404
136+
2024-07-30,3,503
137+
2024-07-31,212,200
138+
2024-07-31,11,404
139+
2024-07-31,7,503
140+
2024-08-01,206,200
141+
2024-08-01,13,503
142+
2024-08-01,11,404
143+
2024-08-02,216,200
144+
2024-08-02,7,404
145+
2024-08-02,7,503
146+
2024-08-03,214,200
147+
2024-08-03,14,404
148+
2024-08-03,2,503
149+
2024-08-04,213,200
150+
2024-08-04,13,503
151+
2024-08-04,4,404
152+
2024-08-05,212,200
153+
2024-08-05,9,404
154+
2024-08-05,9,503
155+
2024-08-06,211,200
156+
2024-08-06,11,503
157+
2024-08-06,8,404
158+
2024-08-07,210,200
159+
2024-08-07,13,404
160+
2024-08-07,7,503
161+
2024-08-08,206,200
162+
2024-08-08,16,404
163+
2024-08-08,8,503
164+
2024-08-09,211,200
165+
2024-08-09,13,404
166+
2024-08-09,6,503
167+
2024-08-10,212,200
168+
2024-08-10,13,404
169+
2024-08-10,5,503
170+
2024-08-11,208,200
171+
2024-08-11,12,404
172+
2024-08-11,10,503
173+
2024-08-12,210,200
174+
2024-08-12,10,404
175+
2024-08-12,10,503
176+
2024-08-13,213,200
177+
2024-08-13,9,503
178+
2024-08-13,8,404
179+
2024-08-14,210,200
180+
2024-08-14,13,404
181+
2024-08-14,7,503
182+
2024-08-15,194,200
183+
2024-08-15,8,404
184+
2024-08-15,3,503

0 commit comments

Comments
 (0)