Skip to content

Commit 15bfc41

Browse files
smoorssmoors
authored
Merge pull request #19 from wpoely86/pixiu_alerts
Fix alerts for Pixiu + extend with logins AB#18742
2 parents e2bb84d + df9f2b6 commit 15bfc41

File tree

3 files changed

+63
-15
lines changed

3 files changed

+63
-15
lines changed

‎.github/workflows/testpatterns.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -6,21 +6,21 @@ jobs:
66
runs-on: ubuntu-latest
77
steps:
88
- name: add logstash repo
9-
uses: myci-actions/add-deb-repo@4
9+
uses: myci-actions/add-deb-repo@master
1010
with:
1111
repo: deb https://artifacts.elastic.co/packages/8.x/apt stable main
1212
repo-name: elastic-8.x
13-
keys: D27D666CD88E42B4
13+
keys-asc: https://artifacts.elastic.co/GPG-KEY-elasticsearch
1414
- name: install logstash
1515
run: sudo apt-get install logstash
1616
- name: add logstash to path
1717
run: echo "/usr/share/logstash/bin" >> $GITHUB_PATH
1818
- name: fix directory permissions of logstash data directory
1919
run: sudo chmod 777 /usr/share/logstash/data
2020
- name: Checkout code
21-
uses: actions/checkout@v3
21+
uses: actions/checkout@v4
2222
- name: Set up Python
23-
uses: actions/setup-python@v4
23+
uses: actions/setup-python@v5
2424
with:
2525
python-version: '3.9'
2626
- name: Install dependencies

‎files/pixiu

+9-5
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,22 @@ PIXIU_NGINX \s*%{IPORHOST:syslog_hostname} %{SYSLOGPROG} %{IPORHOST:clientip} -
66

77
PIXIU_TOMCAT \s*%{IPORHOST:syslog_hostname} %{SYSLOGPROG} \[%{HTTPDATE:timestamp}\]\^%{IPORHOST:clientip}\^%{HTTPDUSER:auth}\^%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\^%{GREEDYDATA:other}
88

9-
PIXIU_ALARM_PREFIX \s*%{IPORHOST:syslog_hostname} %{SYSLOGPROG}: .+%{IPORHOST} (?<alarm_object>OceanStor-Distributed-Storage)\s+%{NUMBER} (?<alarm_id>0x[A-F0-9]+) (?<alarm_severity>Major).*:
9+
PIXIU_ALARM_PREFIX \s*%{IPORHOST:syslog_hostname} %{SYSLOGPROG}: .+%{IPORHOST} (?<alarm_object>OceanStor-Distributed-Storage)\s+%{NUMBER} (?<alarm_id>0x[A-F0-9]+) (?<alarm_severity>%{WORD}).*:
1010

1111
PIXIU_ALARM_USER_TYPE user name, owner name %{USERNAME:username}
1212

1313
PIXIU_BYTES [0-9]+(?:K|M|G|T|P)?B
1414

1515
PIXIU_QUOTA_SIZE_TYPE (?:(?<used_files>%{NUMBER:int})|(?<used_space>%{PIXIU_BYTES}))
1616

17-
PIXIU_ALARM_COMMON The used (?<quota_used_type>%{WORD}) \(%{PIXIU_QUOTA_SIZE_TYPE}\) of (?:the )?quota \(type (?<quota_type>%{WORD}) quota(?:, %{PIXIU_ALARM_USER_TYPE})?\) of dtree \(name (?<dtree>%{WORD}), ID (?<dtree_id>%{NUMBER:int})\) in namespace \(name (?<namespace>%{WORD}), ID (?<namespace_id>%{NUMBER:int})\)
17+
PIXIU_ALARM_COMMON The used (?<quota_used_type>%{WORD}) \(%{PIXIU_QUOTA_SIZE_TYPE}\) of (?:the )?quota \(type (?<quota_type>%{WORD}) quota(?:, %{PIXIU_ALARM_USER_TYPE})?\) of dtree \(name (?<dtree>%{WORD}), ID (?<dtree_id>%{NUMBER:int})\) at %{UNIXPATH:path} in namespace \(name (?<namespace>%{WORD}), ID (?<namespace_id>%{NUMBER:int})\)
1818

19-
PIXIU_USER_INODE_ALARM %{PIXIU_ALARM_PREFIX} %{PIXIU_ALARM_COMMON} reaches (?:or approaches )?to the (:?(?<quota_limit_type>%{WORD}) (?:file|space) quantity quota of|(?:file|space) quantity (?<quota_limit_type>%{WORD}) quota of) \((?<quota_files_limit>%{NUMBER:int})\).
19+
PIXIU_USER_INODE_ALARM %{PIXIU_ALARM_PREFIX} %{PIXIU_ALARM_COMMON} reaches (?:or approaches )?to the (:?(?<quota_limit_type>%{WORD}) (?:file|space) quantity quota of|(?:file|space) quantity (?<quota_limit_type>%{WORD}) quota of) \((?<quota_files_limit>%{NUMBER:int})\)\..*
2020

21-
PIXIU_USER_SPACE_ALARM %{PIXIU_ALARM_PREFIX} %{PIXIU_ALARM_COMMON} reaches (?:or approaches )?to the (:?(?<quota_limit_type>%{WORD}) space(?: quantity)? quota of|space(?: quantity)? (?<quota_limit_type>%{WORD}) quota of) \((?<quota_space_limit>%{PIXIU_BYTES})\).
21+
PIXIU_USER_SPACE_ALARM %{PIXIU_ALARM_PREFIX} %{PIXIU_ALARM_COMMON} reaches (?:or approaches )?to the (:?(?<quota_limit_type>%{WORD}) space(?: quantity)? quota of|space(?: quantity)? (?<quota_limit_type>%{WORD}) quota of) \((?<quota_space_limit>%{PIXIU_BYTES})\)\..*
2222

23-
PIXIU_ALARM %{PIXIU_USER_INODE_ALARM}|%{PIXIU_USER_SPACE_ALARM}
23+
PIXIU_LOGIN_STATUS failed|succeeded
24+
25+
PIXIU_LOGIN_ALARM %{PIXIU_ALARM_PREFIX} User \(user name %{USERNAME:username}\) %{PIXIU_LOGIN_STATUS:state} (?:to log in|in logging in) from source \(%{IPORHOST:source_ip}\)\..*
26+
27+
PIXIU_ALARM %{PIXIU_USER_INODE_ALARM}|%{PIXIU_USER_SPACE_ALARM}|%{PIXIU_LOGIN_ALARM}

‎tests/data/pixiu

+50-6
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ data = [
5858
},
5959
},
6060
{
61-
"raw": "<187>Jul 19 14:22:39 C4STO01-Node1 alarm[2613490]: <186>2023-07-19 14:11:57 DST 2023-07-19 14:22:37 DST 172.19.104.10 OceanStor-Distributed-Storage 216045 0xFEA6A000F Major(2): The used files (39293) of quota (type user quota, user name, owner name vsc10800) of dtree (name 108, ID 4137) in namespace (name data, ID 69) reaches to the file quantity soft quota of (38095).",
61+
"raw": "<187>Jul 19 14:22:39 C4STO01-Node1 alarm[2613490]: <186>2023-07-19 14:11:57 DST 2023-07-19 14:22:37 DST 172.19.104.10 OceanStor-Distributed-Storage 216045 0xFEA6A000F Major(2): The used files (39293) of quota (type user quota, user name, owner name vsc10800) of dtree (name 108, ID 4137) at /108 in namespace (name data, ID 69) reaches to the file quantity soft quota of (38095).",
6262
"expected": {
6363
"@source_host": "C4STO01-Node1",
6464
"program": "alarm",
@@ -75,10 +75,11 @@ data = [
7575
"namespace_id": 69,
7676
"quota_limit_type": "soft",
7777
"quota_files_limit": 38095,
78+
"path": "/108",
7879
},
7980
},
8081
{
81-
"raw": "<187>Jul 19 14:22:39 C4STO01-Node1 alarm[2613490]: <186>2023-07-19 14:06:48 DST 2023-07-19 14:22:37 DST 172.19.104.10 OceanStor-Distributed-Storage 216042 0xFEA6A0011 Major(2): The used files (40000) of the quota (type user quota, user name, owner name vsc10800) of dtree (name 108, ID 4137) in namespace (name data, ID 69) reaches or approaches to the hard file quantity quota of (40000). The user or user group will fail to write data.",
82+
"raw": "<187>Jul 19 14:22:39 C4STO01-Node1 alarm[2613490]: <186>2023-07-19 14:06:48 DST 2023-07-19 14:22:37 DST 172.19.104.10 OceanStor-Distributed-Storage 216042 0xFEA6A0011 Major(2): The used files (40000) of the quota (type user quota, user name, owner name vsc10800) of dtree (name 108, ID 4137) at /108 in namespace (name data, ID 69) reaches or approaches to the hard file quantity quota of (40000). The user or user group will fail to write data.",
8283
"expected": {
8384
"@source_host": "C4STO01-Node1",
8485
"program": "alarm",
@@ -95,10 +96,11 @@ data = [
9596
"namespace_id": 69,
9697
"quota_limit_type": "hard",
9798
"quota_files_limit": 40000,
99+
"path": "/108",
98100
},
99101
},
100102
{
101-
"raw": "<187>Aug 20 20:18:25 C4STO01-Node1 alarm[2613490]: <186>2023-08-20 20:18:25 DST 172.19.104.10 OceanStor-Distributed-Storage 259068 0xFEA6A000E Major(1): The used files (1501) of quota (type directory quota) of dtree (name test1, ID 4101) in namespace (name admin, ID 71) reaches to the file quantity soft quota of (1000).",
103+
"raw": "<187>Aug 20 20:18:25 C4STO01-Node1 alarm[2613490]: <186>2023-08-20 20:18:25 DST 172.19.104.10 OceanStor-Distributed-Storage 259068 0xFEA6A000E Major(1): The used files (1501) of quota (type directory quota) of dtree (name test1, ID 4101) at / in namespace (name admin, ID 71) reaches to the file quantity soft quota of (1000).",
102104
"expected": {
103105
"@source_host": "C4STO01-Node1",
104106
"program": "alarm",
@@ -114,10 +116,11 @@ data = [
114116
"namespace_id": 71,
115117
"quota_limit_type": "soft",
116118
"quota_files_limit": 1000,
119+
"path": "/",
117120
},
118121
},
119122
{
120-
"raw": "<187>Aug 20 20:16:52 C4STO01-Node1 alarm[2613490]: <186>2023-08-20 20:16:50 DST 172.19.104.10 OceanStor-Distributed-Storage 259067 0xFEA6A000B Major(1): The used files (2000) of quota (type directory quota) of dtree (name test1, ID 4101) in namespace (name admin, ID 71) reaches or approaches to the file quantity hard quota of (2000). Data will fail to be written.",
123+
"raw": "<187>Aug 20 20:16:52 C4STO01-Node1 alarm[2613490]: <186>2023-08-20 20:16:50 DST 172.19.104.10 OceanStor-Distributed-Storage 259067 0xFEA6A000B Major(1): The used files (2000) of quota (type directory quota) of dtree (name test1, ID 4101) at / in namespace (name admin, ID 71) reaches or approaches to the file quantity hard quota of (2000). Data will fail to be written.",
121124
"expected": {
122125
"@source_host": "C4STO01-Node1",
123126
"program": "alarm",
@@ -133,10 +136,11 @@ data = [
133136
"namespace_id": 71,
134137
"quota_limit_type": "hard",
135138
"quota_files_limit": 2000,
139+
"path": "/",
136140
},
137141
},
138142
{
139-
"raw": "<187>Aug 21 09:56:38 C4STO01-Node1 alarm[2613490]: <186>2023-08-21 09:56:37 DST 172.19.104.10 OceanStor-Distributed-Storage 260204 0xFEA6A000B Major(1): The used space (1024MB) of quota (type directory quota) of dtree (name test1, ID 4101) in namespace (name admin, ID 71) reaches or approaches to the space hard quota of (1024MB). Data will fail to be written.",
143+
"raw": "<187>Aug 21 09:56:38 C4STO01-Node1 alarm[2613490]: <186>2023-08-21 09:56:37 DST 172.19.104.10 OceanStor-Distributed-Storage 260204 0xFEA6A000B Major(1): The used space (1024MB) of quota (type directory quota) of dtree (name test1, ID 4101) at / in namespace (name admin, ID 71) reaches or approaches to the space hard quota of (1024MB). Data will fail to be written.",
140144
"expected": {
141145
"@source_host": "C4STO01-Node1",
142146
"program": "alarm",
@@ -152,10 +156,11 @@ data = [
152156
"namespace_id": 71,
153157
"quota_limit_type": "hard",
154158
"quota_space_limit": 1024000000,
159+
"path": "/",
155160
},
156161
},
157162
{
158-
"raw": "<187>Aug 20 19:40:26 C4STO01-Node1 alarm[2613490]: <186>2023-08-20 19:32:24 DST 2023-08-20 19:40:26 DST 172.19.104.10 OceanStor-Distributed-Storage 259045 0xFEA6A000F Major(2): The used space (6638MB) of quota (type user quota, user name, owner name vsc10042) of dtree (name 100, ID 8193) in namespace (name user, ID 70) reaches to the space soft quota of (6144MB).",
163+
"raw": "<187>Aug 20 19:40:26 C4STO01-Node1 alarm[2613490]: <186>2023-08-20 19:32:24 DST 2023-08-20 19:40:26 DST 172.19.104.10 OceanStor-Distributed-Storage 259045 0xFEA6A000F Major(2): The used space (6638MB) of quota (type user quota, user name, owner name vsc10042) of dtree (name 100, ID 8193) at /100 in namespace (name user, ID 70) reaches to the space soft quota of (6144MB).",
159164
"expected": {
160165
"@source_host": "C4STO01-Node1",
161166
"program": "alarm",
@@ -172,6 +177,45 @@ data = [
172177
"namespace_id": 70,
173178
"quota_limit_type": "soft",
174179
"quota_space_limit": 6144000000,
180+
"path": "/100",
181+
},
182+
},
183+
{
184+
"raw": "<190>Feb 27 12:04:21 HKSTO03-Node1 alarm[1266782]: <189>2024-02-27 12:04:21 172.19.96.130 OceanStor-Distributed-Storage 543842 0x200F002A0015 Informational(8): User (user name adm_wpoelmans) succeeded in logging in from source (172.18.124.113).",
185+
"expected": {
186+
"@source_host": "HKSTO03-Node1",
187+
"program": "alarm",
188+
"username": "adm_wpoelmans",
189+
"source_ip": "172.18.124.113",
190+
"state": "succeeded",
191+
},
192+
},
193+
{
194+
"raw": "<190>Feb 27 12:00:46 C4STO01-Node1 alarm[76472]: <189>2024-02-27 12:00:46 172.19.104.10 OceanStor-Distributed-Storage 753861 0x200F002A0015 Informational(8): User (user name admin) succeeded in logging in from source (172.18.124.113).",
195+
"expected": {
196+
"@source_host": "C4STO01-Node1",
197+
"program": "alarm",
198+
"username": "admin",
199+
"source_ip": "172.18.124.113",
200+
"state": "succeeded",
201+
},
202+
},
203+
{
204+
"raw": "<190>Feb 27 12:09:52 HKSTO03-Node1 alarm[1266782]: <189>2024-02-27 12:09:52 172.19.96.130 OceanStor-Distributed-Storage 543846 0x200F002A0016 Informational(8): User (user name bsdfsdf) failed to log in from source (172.18.124.113). Error code: 0x40403281.",
205+
"expected": {
206+
"@source_host": "HKSTO03-Node1",
207+
"program": "alarm",
208+
"username": "bsdfsdf",
209+
"source_ip": "172.18.124.113",
210+
"state": "failed",
211+
},
212+
},
213+
{
214+
"raw": "<187>May 30 00:20:05 C4STO01-Node1 alarm[4162768]: <186>2024-05-29 23:52:06 DST 2024-05-30 00:20:04 DST 172.19.104.10 OceanStor-Distributed-Storage 1105311 0xFEA6A0011 Major(2): The used files (20000) of the quota (type user quota, user name, owner name vsc10816) of dtree (name 108, ID 8200) at /108 in namespace (name user, ID 70) reaches or approaches to the hard file quantity quota of (20000). The user or user group will fail to write data.",
215+
"expected": {
216+
"program": "alarm",
217+
"quota_used_type": "files",
218+
"path": "/108",
175219
},
176220
},
177221
]

0 commit comments

Comments
 (0)