Skip to content

Commit 7194835

Browse files
HofiOneHofiOne
authored
Added file() and wildcard-file() following explanations (#102)
Added the new monitor-freq() option description Documentation of syslog-ng/syslog-ng#4998 Signed-off-by: Hofi [[email protected]](mailto:[email protected])
2 parents 8555e8b + 742e1da commit 7194835

File tree

10 files changed

+212
-11
lines changed

10 files changed

+212
-11
lines changed

‎_data/link_aliases.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
#
2+
# Enclose the `aliases` strings inside / and / to use ruby regexp.
3+
# Any not enclosed strings will be matched exactly like written.
4+
#
15

26
adm-temp-macro-ose:
37
aliases: [ "/[Mm]acros/" ]
@@ -94,3 +98,9 @@ adm-about-glossary#bsd-syslog-protocol:
9498

9599
adm-about-glossary#ietf-syslog-protocol:
96100
aliases: [ "IETF-syslog protocol", "IETF-syslog" ]
101+
102+
adm-src-wild:
103+
aliases: [ "/wildcard-file(?:\\(\\))? source[s]?/" ]
104+
105+
adm-src-file:
106+
aliases: [ "/file(?:\\(\\))? source[s]?/" ]

‎_data/navigation.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -142,8 +142,10 @@ admin-guide-nav:
142142
subnav:
143143
- title: "Notes on reading kernel messages"
144144
url: /admin-guide/060_Sources/020_File/000_Notes_on_reading_kernel_messages
145+
- title: "How file sources are followed"
146+
url: /admin-guide/060_Sources/020_File/001_File_following
145147
- title: "file() source options"
146-
url: /admin-guide/060_Sources/020_File/001_File_source_options
148+
url: /admin-guide/060_Sources/020_File/002_File_source_options
147149
- title: "Arr logs"
148150
url: /admin-guide/060_Sources/025_Arr_logs/README
149151
subnav:

‎_includes/doc/admin-guide/options/follow-freq.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,10 @@
55

66
*Description:* Indicates that the source should be checked periodically.
77
This is useful for files which always indicate readability, even though
8-
no new lines were appended. If this value is higher than zero, syslog-ng
9-
will not attempt to use poll() on the file, but checks whether the file
10-
changed every time the follow-freq() interval (in seconds) has elapsed.
8+
no new lines were appended (e.g. regular file system files). If this value
9+
is higher than zero, syslog-ng will not attempt to use ivykis file change
10+
notification methods on the file (poll(), epoll(), etc.), but checks whether
11+
the file changed every time the follow-freq() interval (in seconds) has elapsed.\
1112
Floating-point numbers (for example, **1.5**) can be used as well.
13+
14+
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}

‎_includes/doc/admin-guide/warnings/file-source-follow-warning.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), the monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
2+
{: .notice--warning}

‎_sass/minimal-mistakes/minimal-mistakes/_tables.scss

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ th {
3030
td {
3131
padding: 0.5em;
3232
border-bottom: 1px solid mix(#000, $border-color, 25%);
33+
&.right-edged-col {
34+
border-right: 1px solid mix(#000, $border-color, 25%);
35+
}
3336
}
3437

3538
tr,

‎doc/_admin-guide/060_Sources/020_File/001_File_following.md

Lines changed: 164 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
---
2+
title: How content changes are followed in file() and wildcard-file() sources
3+
short_title: How file sources are followed
4+
id: adm-src-wild-follow
5+
description: >-
6+
The {{ site.product.short_name }} application can use various methods to detect changes
7+
in the followed [[file()|adm-src-file]] and wildcard-file() sources. Under the hood, two different change
8+
watchers operate: one for detecting changes in the followed directories and one for
9+
monitoring changes in the watched files.
10+
---
11+
12+
The detection method for directory changes can be controlled via monitor-method(). If the method is `poll`, {{ site.product.short_name }} will set up an (ivykis) timer that periodically checks and compares the content of the given directory at the frequency specified by monitor-freq(). This periodic polling can be resource-intensive (mainly CPU), so selecting the proper value for monitor-freq() is important. This is not an issue on Linux, where you can use `inotify` for monitor-method(), which automatically notifies {{ site.product.short_name }} about changes with no significant resource usage impact.
13+
14+
**NOTE:** For macOS and FreeBSD, a `kqueue` based implementation will be available in the near future, offering similar performance to `inotify` on Linux. Stay tuned!
15+
{: .notice--info}
16+
17+
Detecting file content changes involves more factors that can affect resource usage and overall performance. {{ site.product.short_name }} uses two main methods to monitor file content changes.
18+
19+
The first method is automatically selected if the follow-freq() option has a value greater than 0. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires. Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
20+
21+
The second method is activated if the follow-freq() option is set to 0. It uses ivykis poll methods, sometimes resembling the poll method for directory change watching described above (with its performance penalties), but often working similarly to the `inotify` version (with seamless performance).
22+
23+
The following table shows which method is selected in different cases.
24+
25+
<table border="0" cellspacing="0" cellpadding="0" width="1397">
26+
<tr>
27+
<td width="99" class="right-edged-col">
28+
<p align="center">follow-freq()</p>
29+
</td>
30+
<td width="200" colspan="2" class="right-edged-col">
31+
<p align="center">file follow method</p>
32+
</td>
33+
<td width="151" class="right-edged-col">
34+
<p align="center">linux</p>
35+
</td>
36+
<td width="152" class="right-edged-col">
37+
<p align="center">macOS, FreeBSD</p>
38+
</td>
39+
<td width="153">
40+
<p align="center">Solaris</p>
41+
</td>
42+
</tr>
43+
<tr>
44+
<td width="99" rowspan="6" class="right-edged-col">
45+
<p align="center">0</p>
46+
</td>
47+
<td width="150" rowspan="6" class="right-edged-col">
48+
<p align="center">ivykis poll</p>
49+
</td>
50+
<td width="152" class="right-edged-col">
51+
<p align="center">port-timer, port</p>
52+
</td>
53+
<td width="151" class="right-edged-col">
54+
<p align="center">n.a.</p>
55+
</td>
56+
<td width="152" class="right-edged-col">
57+
<p align="center">n.a.</p>
58+
</td>
59+
<td width="153">
60+
<p align="center">should work<br>(not tested yet)</p>
61+
</td>
62+
</tr>
63+
<tr>
64+
<td width="152" class="right-edged-col">
65+
<p align="center">dev-poll</p>
66+
</td>
67+
<td width="151" class="right-edged-col">
68+
<p align="center">n.a.</p>
69+
</td>
70+
<td width="152" class="right-edged-col">
71+
<p align="center">n.a.</p>
72+
</td>
73+
<td width="153">
74+
<p align="center">should work<br>(not tested yet)</p>
75+
</td>
76+
</tr>
77+
<tr>
78+
<td width="152" class="right-edged-col">
79+
<p align="center">epoll-timer, epoll</p>
80+
</td>
81+
<td width="151" class="right-edged-col">
82+
<p align="center">does not work </p>
83+
</td>
84+
<td width="152" class="right-edged-col">
85+
<p align="center">n.a.</p>
86+
</td>
87+
<td width="153">
88+
<p align="center">should work<br>(not tested yet)</p>
89+
</td>
90+
</tr>
91+
<tr>
92+
<td width="152" class="right-edged-col">
93+
<p align="center">kqueue</p>
94+
</td>
95+
<td width="151" class="right-edged-col">
96+
<p align="center">n.a.</p>
97+
</td>
98+
<td width="152" class="right-edged-col">
99+
<p align="center">works</p>
100+
</td>
101+
<td width="153">
102+
<p align="center">n.a.</p>
103+
</td>
104+
</tr>
105+
<tr>
106+
<td width="152" class="right-edged-col">
107+
<p align="center">ppoll</p>
108+
</td>
109+
<td width="151" class="right-edged-col">
110+
<p align="center">works, but always signals readability</p>
111+
</td>
112+
<td width="152" class="right-edged-col">
113+
<p align="center">n.a.</p>
114+
</td>
115+
<td width="153">
116+
<p align="center">should work<br>(not tested yet)</p>
117+
</td>
118+
</tr>
119+
<tr>
120+
<td width="152" class="right-edged-col">
121+
<p align="center">poll</p>
122+
</td>
123+
<td width="151" class="right-edged-col">
124+
<p align="center">works, but always signals readability</p>
125+
</td>
126+
<td width="152" class="right-edged-col">
127+
<p align="center">works, but always signals readability</p>
128+
</td>
129+
<td width="153">
130+
<p align="center">works, but always signals readability</p>
131+
</td>
132+
</tr>
133+
<tr>
134+
<td width="99" rowspan="3" class="right-edged-col">
135+
<p align="center">&gt; 0</p>
136+
</td>
137+
<td width="150" colspan="2" rowspan="3" class="right-edged-col">
138+
<p align="center">syslog-ng poll<br>using ivykis timer with timer freq set to follow-freq() value</p>
139+
</td>
140+
<td width="151" rowspan="3" class="right-edged-col">
141+
<p align="center">works</p>
142+
</td>
143+
<td width="152" rowspan="3" class="right-edged-col">
144+
<p align="center">works</p>
145+
</td>
146+
<td width="153">
147+
<p align="center">works</p>
148+
</td>
149+
</tr>
150+
</table>
151+
152+
A bit more detail about the notation in the platform columns and what they really mean:
153+
154+
`n.a.` - Means that the feature is not supported on the given platform by default, which has a significant impact on how the final ivykis poll method is selected. Ivykis tries to set up (at initialization time) the method to be used in the order enumerated in the table above. If an option is `n.a.` (determined at build time), then the next option will be used automatically. The first available option will be used, and if it does not work on the given platform (see `does not work` below), then {{ site.product.short_name }} will stop that file source with an error. This can be controlled using the `IV_EXCLUDE_POLL_METHOD` environment variable. Methods enumerated in it will be excluded from the ivykis initialization flow, and the next available (and not excluded) one will be used. The strings that can be used in `IV_EXCLUDE_POLL_METHOD` are `port-timer port dev_poll epoll-timerfd epoll kqueue ppoll poll` in the same order as in the table.\
155+
e.g., on Linux you should use `IV_EXCLUDE_POLL_METHOD="epoll-timerfd epoll"` to force the usage of the `ppoll` method, as `port-timer port dev_poll` are not available, and `epoll-timerfd epoll` are not working currently. However, note that currently `ppoll` and `poll` are the only working options on Linux, and they are far from optimal, unlike on BSD-based systems like macOS, where the default `kqueue` is a perfect option to use.
156+
157+
**NOTE:** We are planning to add an `inotify` or similar-based solution to ivykis that could perform similarly to `kqueue`. Stay tuned!
158+
{: .notice--info}
159+
160+
`works` - Means it is tested and works seamlessly (based on our tests).
161+
162+
`works, but always signals readability` - Means that the method is available on the given platform, but it is primarily designed for sockets, pipes, and similar uses, not for regular files. For regular files, it is always triggered (because regular files are always readable), behaving similarly to the `poll` method of directory monitoring with all of its disadvantages. Moreover, it could lead to even higher resource consumption (mainly CPU load) because follow-freq() does not control the frequency of the triggered internal file checks, which could occur hundreds or thousands of times per second.
163+
164+
`does not work` - Means that the method is available on the given platform but currently does not work as expected for various reasons.

‎doc/_admin-guide/060_Sources/020_File/001_File_source_options.md renamed to ‎doc/_admin-guide/060_Sources/020_File/002_File_source_options.md

File renamed without changes.

‎doc/_admin-guide/060_Sources/020_File/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ The file driver has a single required parameter specifying the file to
1717
open. If you want to use wildcards in the filename, use the [[wildcard-file() source|adm-src-wild]].
1818
For the list of available optional parameters, see file() source options.
1919

20+
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}
21+
2022
**Declaration**
2123

2224
```config

‎doc/_admin-guide/060_Sources/030_Wildcard-file/000_Wildcard-file_options.md

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -105,16 +105,28 @@ monitor. If the wildcard-file source matches more files than the value
105105
of the max-files() option, it is random which files will {{ site.product.short_name }}
106106
actually monitor. The default value of max-files() is 100.
107107

108+
## monitor-freq()
109+
110+
|Type: | number |
111+
|Default: | value of follow-freq() |
112+
113+
*Description:* Indicates how frequently changes to the source file creation, move, or deletion should be checked if monitor-method() is set to `poll`.\
114+
Floating-point numbers (for example, **1.5**) can be used as well. Please note, for backward compatibility reasons, the default value of monitor-freq() is equal to the value of follow-freq().
115+
116+
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}
117+
108118
## monitor-method()
109119

110-
|Accepted values:| auto \| inotify \| poll|
111-
|Default:| auto|
120+
|Accepted values:| auto \| inotify \| poll|
121+
|Default: | auto |
112122

113123
*Description:* If the platform supports inotify, {{ site.product.short_name }} uses it
114-
automatically to detect changes to the source files. If inotify is not
115-
available, {{ site.product.short_name }} polls the files as set in the follow-freq()
116-
option. To force {{ site.product.short_name }} poll the files even if inotify is
117-
available, set this option to **poll**.
124+
automatically to detect creation, move, or deletion of the source files. If inotify is not
125+
available, {{ site.product.short_name }} polls the above mentioned file changes as set in
126+
the monitor-freq() option. To force {{ site.product.short_name }} poll the file changes even if
127+
inotify is available, set this option to **poll**.
128+
129+
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}
118130

119131
{% include doc/admin-guide/options/multi-line-garbage.md %}
120132

@@ -154,8 +166,9 @@ source s_file_subdirectories {
154166
base-dir("/var/log")
155167
filename-pattern("*.log")
156168
recursive(yes)
169+
monitor-freq(0.5)
157170
follow-freq(1)
158-
log-fetch-limit(100)
171+
log-fetch-limit(200)
159172
);
160173
};
161174
```

‎doc/_admin-guide/060_Sources/030_Wildcard-file/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ persist file depends on the package you installed {{ site.product.short_name }}
1717
typically it is /var/lib/syslog-ng/syslog-ng.persist or
1818
/var/lib/syslog-ng/syslog-ng.persist.
1919

20+
{% include doc/admin-guide/warnings/file-source-follow-warning.md %}
21+
2022
**Declaration**
2123

2224
```config

0 commit comments

Comments
 (0)