Skip to content

Commit d5d8c93

Browse files
LucasRoesleralexellis
LucasRoesler
authored and
alexellis
committed
New tutorial for installing openfaas-loki
**What** - Create a new tutorial blog post that shows how to install openfaas, loki, and the openfaas-loki provider using arkade. Signed-off-by: Lucas Roesler <[email protected]>
1 parent 49d693f commit d5d8c93

File tree

3 files changed

+245
-0
lines changed

3 files changed

+245
-0
lines changed

‎_posts/2020-07-06-openfaas-loki-into.md

+245
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
---
2+
title: Time travelling through your OpenFaaS Function logs
3+
description: "In this post Lucas will show you how to query historic logs for your OpenFaaS functions using Grafana Loki and the openfaas-loki log provider."
4+
date: 2020-07-06
5+
image: /images/2020-openfaas-loki/2020-06-loki-logs-timey-wobbly.jpg
6+
categories:
7+
- logs
8+
- kubernetes
9+
- serverless
10+
- tutorial
11+
- grafana
12+
- loki
13+
author_staff_member: lucas
14+
dark_background: true
15+
16+
---
17+
18+
In this post Lucas will show you how to query historic logs for your OpenFaaS functions using Grafana Loki and the [`openfaas-loki`](https://github.com/LucasRoesler/openfaas-loki) [log provider](https://docs.openfaas.com/architecture/logs-provider/).
19+
20+
Logs are critical to observing and debugging systems once in production. OpenFaaS builds in support for logs in _every_ installation. But it can be even better by using a new log provider. In this post we will show you how to supercharge the logs by installing [Grafana's Loki](https://grafana.com/oss/loki) and configure OpenFaaS to use Loki.
21+
22+
## Motivation
23+
24+
In the default installation of OpenFaaS, the logs come from the underlying orchestration tool. This is great, because logs are automatically available without any additional effort, but this means logs only exist for the _current_ function instances. This means that as your function scales from 0 to 100 and then back to 0, you will no long have access to the function logs because the orchestraition system does not keep logs after the function instance is removed! This is simply the default behavior of systems like Docker Swarm and Kubernetes. If a user reports and error, but the function instance has scaled down already, the logs you need to debug the issue are now gone.
25+
26+
Sometimes it's important to retrieve the logs for function invocations from several days or even weeks ago. To achieve that, we need to aggregate, store and index the logs from each function. Traditionally, this has meant installing the [ELK stack](https://www.elastic.co/what-is/elk-stack): ElasticSearch, Logstash, Kibana. This is a great and well established tool; but, in [late 2018 Grafana introduced](https://kccna18.sched.com/event/GrXC/on-the-oss-path-to-full-observability-with-grafana-david-kaltschmidt-grafana-labs) a new alternative aimed squarely at the Cloud Native world: [Loki](https://grafana.com/oss/loki/).
27+
28+
> Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.
29+
30+
Loki is a light-weight tool that's easy to use. When it comes to Kubernetes, deployment is simple through the project's Helm chart.
31+
32+
Once a log aggregator is installed and configured it will often have a UI for exploring the logs, for ELK this is [Kibana](https://www.elastic.co/kibana) and for Loki it is [Grafana Explore](https://grafana.com/docs/grafana/latest/features/explore/).
33+
34+
Since OpenFaaS Gateway 0.15.0+ (July 2019), OpenFaaS supports [pluggable logging providers](https://docs.openfaas.com/architecture/logs-provider/) to support streaming function logs via the Gateway API and the `faas-cli`. With the latest release of [arkade](https://get-arkade.dev/) all three (OpenFaaS, Loki, and Grafana) can be installed in just a few lines, allowing you to access the same function logs from the `faas-cli`, the API, or the Grafana Explore dashboard. We will walk-though the steps below.
35+
36+
37+
## Prerequisites
38+
39+
Before we start we need a couple of tools to help us quickly set up our environment:
40+
41+
* [docker](https://docs.docker.com/get-docker/) - required to use KinD
42+
* [kind](https://kind.sigs.k8s.io/docs/user/quick-start/) - a tool for running local Kubernetes clusters using Docker
43+
* [arkade](https://github.com/alexellis/arkade) - official installer for OpenFaaS, that supports many other applications for Kubernetes
44+
* [faas-cli](https://docs.openfaas.com/cli/install/) - the CLI which communicates with the OpenFaaS gateway
45+
* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) - the CLI which communicates with your Kubernetes cluster
46+
47+
## Prepare environment
48+
49+
Before showing you how to run our functions on a schedule, first we need to set up the environment with the tools mentioned above.
50+
51+
### Create cluster
52+
53+
First thing we need is running Kubernetes cluster:
54+
55+
```bash
56+
kind create cluster
57+
```
58+
59+
Wait for the installation to finish run this command to verify that the cluster is ready:
60+
61+
```bash
62+
kubectl -n kube-system rollout status deployment/coredns
63+
```
64+
65+
### Install OpenFaaS
66+
67+
With `arkade` the installation of OpenFaaS boils down to single line command:
68+
69+
```bash
70+
arkade install openfaas
71+
```
72+
73+
Wait for the OpenFaaS gateway to be ready
74+
75+
```bash
76+
kubectl rollout status -n openfaas deploy/gateway
77+
```
78+
79+
### Install faas-cli
80+
81+
The `faas-cli` can also be installed via `arkade`
82+
83+
```bash
84+
arkade get faas-cli
85+
```
86+
87+
### Deploy a sample function
88+
89+
To test the logs we can use any function from the function store, for example, Cows is a fun function. Note that we need to forward the OpenFaaS API to a local port and login to OpenFaaS first
90+
91+
```bash
92+
kubectl port-forward -n openfaas svc/gateway 8080:8080 &
93+
kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode | faas-cli login --username admin --password-stdin
94+
faas-cli store deploy "ASCII Cows"
95+
96+
faas-cli invoke cows <<< ""
97+
____
98+
(____)
99+
100+
(__)
101+
(oo)
102+
/-------\/
103+
/ | ||
104+
* ||----||
105+
~~ ~~
106+
holy cow
107+
```
108+
109+
We can see the logs powered by Kubernetes using
110+
111+
```bash
112+
faas-cli logs cows
113+
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
114+
Handling connection for 8080
115+
2020-06-27T16:37:13Z 2020/06/27 16:37:13 Version: 0.13.0 SHA: fa93655d90d1518b04e7cfca7d7548d7d133a34e
116+
2020-06-27T16:37:13Z 2020/06/27 16:37:13 Read/write timeout: 5s, 5s. Port: 8080
117+
2020-06-27T16:37:13Z 2020/06/27 16:37:13 Writing lock-file to: /tmp/.lock
118+
2020-06-27T16:37:13Z 2020/06/27 16:37:13 Metrics server. Port: 8081
119+
2020-06-27T16:37:17Z 2020/06/27 16:37:17 Forking fprocess.
120+
2020-06-27T16:37:17Z 2020/06/27 16:37:17 Wrote 119 Bytes - Duration: 0.130287 seconds
121+
```
122+
123+
We will see a slightly different format when we switch to Loki later.
124+
125+
### Install Loki and Grafana
126+
127+
With `arkade`, we need this single line to install both Grafana _and_ Loki:
128+
129+
```bash
130+
arkade install loki --grafana
131+
```
132+
133+
Wait for Loki and Grafana to be ready:
134+
135+
```bash
136+
kubectl rollout status deployment loki-stack-grafana
137+
```
138+
139+
Now get your admin password and forward the Grafana port
140+
```bash
141+
kubectl get secret loki-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
142+
kubectl port-forward service/loki-stack-grafana 3000:80 &
143+
```
144+
145+
Then open `http://localhost:3000` and login using username `admin`.
146+
147+
Loki will automatically start collecting logs from _all_ of the Pods it can sees. Meaning you should see logs immediately. Importantly, this means you can see the logs for all of the OpenFaaS components, like the Gateway.
148+
149+
This [link](http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bapp%3D%5C%22gateway%5C%22%7D%22%7D,%7B%22mode%22:%22Logs%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D) will show you the logs (over the last hour) for the Gateway we just installed!
150+
151+
Loki, when installed in Kubernetes, will automatically create a search index for each of the lables on the Pods it sees. Important, this means the Loki query for the logs of an OpenFaaS function looks like `{faas_function="<name>"}` because OpenFaaS always adds the function name as the label `faas_function`. This [link](http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Loki%22,%7B%22expr%22:%22%7Bfaas_function%3D%5C%22cows%5C%22%7D%22%7D,%7B%22mode%22:%22Logs%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D) will show the logs for the Cows function we installed and invoked.
152+
153+
### Installing `openfaas-loki`
154+
155+
Now let's configure OpenFaaS to fetch its logs from Loki.
156+
157+
```bash
158+
arkade install openfaas-loki
159+
```
160+
161+
We then need to configure the OpenFaaS installation, we can do this with `arkade` using the `--set` flag
162+
163+
```bash
164+
arkade install openfaas --set gateway.logsProviderURL=http://openfaas-loki.default:9191/
165+
```
166+
167+
## Testing the logs
168+
169+
### Recreate the port-forward
170+
Earlier, while we were installing and verifying the system, we ran this command
171+
172+
```bash
173+
kubectl port-forward -n openfaas svc/gateway 8080:8080 &
174+
```
175+
176+
Unfortunately, that connection is no longer valid, because we updated the OpenFaaS installation to configure the logs provider. We need to stop the current `port-forward` command and then start a new one:
177+
178+
```bash
179+
killall kubectl
180+
kubectl port-forward -n openfaas svc/gateway 8080:8080 &
181+
```
182+
183+
### Invoke the function and check the logs
184+
Now we can use the `faas-cli` to invoke and see the logs, this time powered by Loki.
185+
186+
```bash
187+
echo "" | faas-cli invoke cows
188+
___ ___
189+
\ \ \ \
190+
(___) ___\__\__\__\__
191+
(o o) | O O O O O O|
192+
--\ /----+-------------+-------/
193+
| O /
194+
\ /
195+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~s~~~~~~~~~~~
196+
Steerage
197+
```
198+
and
199+
200+
```bash
201+
faas-cli logs cows
202+
2020-06-27T18:24:09Z 2020-06-27T18:24:09.633228Z stderr F 2020/06/27 18:24:09 Forking fprocess.
203+
2020-06-27T18:24:09Z 2020-06-27T18:24:09.8101492Z stderr F 2020/06/27 18:24:09 Wrote 279 Bytes - Duration: 0.176972 seconds
204+
```
205+
206+
207+
To compare with the Kubernetes powered logs
208+
209+
```bash
210+
2020-06-27T16:37:17Z 2020/06/27 16:37:17 Forking fprocess.
211+
2020-06-27T16:37:17Z 2020/06/27 16:37:17 Wrote 119 Bytes - Duration: 0.130287 seconds
212+
```
213+
214+
We see a slightly different format that is coming from the way Loki scrapes the logs. Primarily
215+
216+
```
217+
2020-06-27T18:24:09.633228Z stderr F
218+
```
219+
220+
This shows the timestamp for when the log was collected and that it came from `stderr` instead of `stdout`.
221+
222+
We can also demonstrate that the logs are now independently stored from the function instance by scaling the function to 0 and still retrieving the same logs
223+
224+
```bash
225+
kubectl -n openfaas-fn scale deployment --replicas 0 cows
226+
faas-cli logs cows
227+
2020-06-27T18:24:09Z 2020-06-27T18:24:09.633228Z stderr F 2020/06/27 18:24:09 Forking fprocess.
228+
2020-06-27T18:24:09Z 2020-06-27T18:24:09.8101492Z stderr F 2020/06/27 18:24:09 Wrote 279 Bytes - Duration: 0.176972 seconds
229+
```
230+
231+
>Note: The `faas-cli logs` defaults to the last 5 mins of logs. If you walk through this tutorial and have a long pause between steps, it is possible that your logs will be empty because of the gap between your invocation and the request. Just run the invoke command again and then get the logs.
232+
> You can extend the time frame using the `--since` flag, for example: `faas-cli logs cows --since=10m`
233+
234+
235+
## Wrapping up
236+
237+
In this post we have shown how you can use Loki and OpenFaaS together so that your function logs are stored independently from the instances. The combination of Loki and Grafana make it even easier to debug functions than before. You can even create dashboards that combine function metrics and logs in one place by using the [new logs panel](https://grafana.com/docs/grafana/latest/panels/visualizations/logs-panel/)
238+
239+
![grafana-logs-dashboard-panel](/images/2020-openfaas-loki/grafana-logs-dashboard.jpg)
240+
241+
If you have other ideas or hacks with Grafana, or would like to help us add another logs provider for your favorite aggregator (e.g. ELK), please connect with the community.
242+
243+
* Join us on [on Slack](https://slack.openfaas.io/) to chat more
244+
* Read [the docs](https://docs.openfaas.com/architecture/logs-provider/) on the Logs Provider interface and checkout [the example in the provider repo](https://github.com/openfaas/faas-provider/tree/master/logs)
245+
* If you have any suggests or issues with Loki let us know on the [`openfaas-loki` repo](https://github.com/LucasRoesler/openfaas-loki)

‎images/2020-openfaas-loki/2020-06-loki-logs-timey-wobbly.jpg

229 KB
Loading

‎images/2020-openfaas-loki/grafana-logs-dashboard.jpg

279 KB
Loading

0 commit comments

Comments
 (0)