-
|
I'm experimenting some issues with the rotation of logs on version 5.6.0. The main log seems to have two rotations. None of them is actually working well and one of them is filling the storage on some devices. On the one hand, logrotate configurationIt is effectively compressing the log each day (or at 10 MB) and never exceeds the quantity of seven files. However, in some devices, the rotation doesn't seem to be working properly. At some point the compressed files stop to generate or are generated on a random time pattern: 
There are indeed enough logs to justify a compressing, because the other rotation seems to be doing its thing: 
In other devices it works perfectly: 
On the other hand, there are a second rotation on play that creates compressed files with the name log4j.xml configuration<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" strict="true" name="KuraConfig" monitorInterval="30">
<Properties>
<Property name="log_dir">/var/log</Property>
<Property name="log_name">kura</Property>
</Properties>
<Filter type="ThresholdFilter" level="trace"/>
<Appenders>
<RollingFile name="RollingFile" fileName="${log_dir}/${log_name}.log" filePattern="${log_dir}/${log_name}-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz">
<PatternLayout>
<Pattern>%d{ISO8601} [%t] %-5p %c{1.} - %m%n%throwable{full}</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="20 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
<RollingFile name="audit" fileName="${log_dir}/${log_name}-audit.log" filePattern="${log_dir}/${log_name}-audit-%d{MM-dd-yy-HH-mm-ss}-%i.log.gz">
<RFC5424Layout enterpriseNumber="28392" includeMDC="true" mdcId="RequestContext" appName="EclipseKura" mdcPrefix="" newLine="true">
<LoggerFields>
<KeyValuePair key="thread" value="%t"/>
<KeyValuePair key="priority" value="%p"/>
<KeyValuePair key="category" value="%c"/>
<KeyValuePair key="exception" value="%ex"/>
</LoggerFields>
</RFC5424Layout>
<Policies>
<SizeBasedTriggeringPolicy size="20 MB"/>
</Policies>
<DefaultRolloverStrategy max="10"/>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="org.eclipse" level="info" additivity="false">
<AppenderRef ref="RollingFile"/>
</Logger>
<Logger name="AuditLogger" level="trace" additivity="false">
<AppenderRef ref="audit"/>
</Logger>
<Root level="info">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>The 20MB size trigger seems to be working perfectly fine but the rollover is not being done, as this files are stored indefinitely, stacking even hundreds of log files on devices with much activity or much time on production, and leading to other issues: 
Given that the first configuration triggers the compression at 10 MB, I don't even understand how the second configuration can even happen. As said, I'd be interested in having more information about how this is working, why is the rollover managed through two independent configurations and how could I fix the issues I'm experiencing. My final interest would be to modify the config a bit but I don't dare doing it at this moment with this cursed behaviour. |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 19 replies
-
|
Hi @pintify, thanks for reporting! This does indeed look weird, we're looking into it. Meanwhile: is there any difference between the devices running out of space and the one which are working? I would probably look at the |
Beta Was this translation helpful? Give feedback.
-
|
After looking into it and talking with more knowledgable members of the Kura team this is our current assessment of the issue:
We're using both a date and an integer in the file pattern. I don't think log4j is happy about it. To fix the issue I think you can either:
The correct way to fix the issue is to setup Let me know if this fixes the issue |
Beta Was this translation helpful? Give feedback.
-
|
Alright, I was noticing that the logrotate-kura.conf file was not properly located. Now I see why, hehe. Thanks! Anyway, why setting up an independent cron job? If the purpose is to run rotation more often, why not promote logrotate from cron.daily to cron.hourly? I guess you have some use case where running every five minutes is needed? I've been checking multiple devices and their logrotate-kura.status file and in some cases even log4j and logrotate get confused with each other (not the most common thing, though): On other cases, the logrotate is installed, running and up to date, but not working properly: user@pc:/var/log$ logrotate -v
logrotate 3.21.0 - Copyright (C) 1995-2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU General Public License
Usage: logrotate [-dfv?] [-d|--debug] [-f|--force] [-m|--mail=command] [-s|--state=statefile] [--skip-state-lock] [--wait-for-state-lock] [-v|--verbose] [-l|--log=logfile] [--version]
[-?|--help] [--usage] [OPTION...] <configfile>
user@pc:/var/log$ ls | grep "kura.log"
kura.log
kura.log-20250611-1749624901.gz
kura.log-20250612-1749686401.gz
kura.log-20250613-1749772801.gz
kura.log-20250614-1749859201.gz
kura.log-20250615-1749945601.gz
kura.log-20250616-1750032001.gz
user@pc:/var/log$ date
Fri Aug 29 10:42:02 UTC 2025
user@pc:/var/log$ cat logrotate-kura.status
logrotate state -- version 2
"/var/log/kura.log" 2025-6-16-0:0:1
"/var/log/kura*.log" 2024-10-31-16:0:0
"/var/log/kura-audit.log" 2025-6-16-12:5:1
"/var/log/kura-console.log" 2025-6-16-9:20:1I have very different versions of logrotate running depending on the device: 3.11.0, 3.19.0 and 3.21.0. But as far as I've seen, all the directives set on the configuration should work on all of them. I keep working on it. |
Beta Was this translation helpful? Give feedback.
-
|
Also, I've noticed two strange things in the logrotate config:
|
Beta Was this translation helpful? Give feedback.
-
|
I was able to reproduce the issue: 
Steps to reproduce: essentially log a large amount of stuff (to trigger the issue Kura needs to write 20MB of logs in less than 5 minutes). I set up a wiregraph in Kura 5.6.0 as follows: 
My current analysis:
@pintify does this make sense to you? Do your devices log a lot of data? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not fully convinced but yes, it could be the reason. Some devices with a normal logging still generate log4j compression files: Example of full /var/log contentGiven the current difference (from 10MB to 20 MB), I don't believe any of our devices fills the log at more rate than 10 MB in five minutes. However, I'll configure some devices with log4j up to 100 MB to test if your hypothesis is correct. |
Beta Was this translation helpful? Give feedback.
-
|
By the way, I think I've found something about logrotate failures. I have two different issues: some devices don't really rotate, and other only do it occasionally. In the first case this is what seems to be happening: Which I except it can be solved applying the same I'll try this to and check again later. Fortunately we have enough devices to test different approaches at the same time. |
Beta Was this translation helpful? Give feedback.
-
|
Hi there @pintify, we created a fix in #5961 that should address the issue. Let me know if it works for you. You can test it by simply overwriting the |
Beta Was this translation helpful? Give feedback.
-
|
That's great news, indeed. I'll apply the configuration in the PR. I guess I should remove also logrotate cron job to ensure there is no conflicting interests. Let me share some issues regarding the tests you proposed. The fix in log4j to ensure rotation doesn't seem to be working properly. It keeps stacking logs despite having removed the %i parameter: 
On the other side, applying Increasing the size of log4j maximum size to 100 MB remains to be challenged yet but in the next week it may be happening. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @pintify, thanks for reporting in.
Can you try out solution implemented in #5962?
I wholeheartedly agree. We're discussing the next steps for a more stable and long-term solution. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @mattdibi, one of the tests I was doing is precisely the #5962 PR. Specifically, I applied the following configuration to log4j new configuration<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright (c) 2018, 2025 Eurotech and/or its affiliates and others
This program and the accompanying materials are made
available under the terms of the Eclipse Public License 2.0
which is available at https://www.eclipse.org/legal/epl-2.0/
SPDX-License-Identifier: EPL-2.0
Contributors:
Eurotech
-->
<Configuration status="warn" strict="true" name="KuraConfig" monitorInterval="30">
<Properties>
<Property name="log_dir">/var/log</Property>
<Property name="log_name">kura</Property>
</Properties>
<Filter type="ThresholdFilter" level="trace"/>
<Appenders>
<Console name="Console" target="SYSTEM_OUT" follow="true">
<PatternLayout>
<Pattern>%d{ISO8601} [%t] %-5p %c{1.} - %m%n%throwable{full}</Pattern>
</PatternLayout>
</Console>
<File name="LoggingFile" fileName="${log_dir}/${log_name}.log">
<PatternLayout>
<Pattern>%d{ISO8601} [%t] %-5p %c{1.} - %m%n%throwable{full}</Pattern>
</PatternLayout>
</File>
<File name="audit" fileName="${log_dir}/${log_name}-audit.log">
<RFC5424Layout enterpriseNumber="28392" includeMDC="true" mdcId="RequestContext" appName="EclipseKura" mdcPrefix="" newLine="true">
<LoggerFields>
<KeyValuePair key="thread" value="%t"/>
<KeyValuePair key="priority" value="%p"/>
<KeyValuePair key="category" value="%c"/>
<KeyValuePair key="exception" value="%ex"/>
</LoggerFields>
</RFC5424Layout>
</File>
</Appenders>
<Loggers>
<Logger name="org.eclipse" level="info" additivity="false">
<AppenderRef ref="Console"/>
<AppenderRef ref="LoggingFile"/>
</Logger>
<Logger name="AuditLogger" level="trace" additivity="false">
<AppenderRef ref="audit"/>
</Logger>
<Root level="info">
<AppenderRef ref="Console"/>
<AppenderRef ref="LoggingFile"/>
</Root>
</Loggers>
</Configuration>Before you comment, I had also commented the |
Beta Was this translation helpful? Give feedback.
-
|
@mattdibi , I've noticed that the new |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Hehe, probably, thanks for the quick reply! I've applied the patch. I'll let you know the next week how it worked out. |
Beta Was this translation helpful? Give feedback.
-
|
In summary, I've successfully achieved a correct log rotation applying:
I've tested the configuration in five different hardware and distributions (all of the Debian based, though) and all of them are working fine now. Other modifications proposed in the discussion did not succeed or they worked only partially. |
Beta Was this translation helpful? Give feedback.
-
The only change that remains to be included is the |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for reporting back your results @pintify, really appreciated 🙏
As noted in my original PR this probably happens because the Closing the discussion for now. |
Beta Was this translation helpful? Give feedback.
-
|
Hello there @pintify, as promised here you can find our long-term solution for the problem. Switching entirely to log4j should allow us to address previously mentioned issues:
Please note that compressed files changed naming convention. |
Beta Was this translation helpful? Give feedback.
-
|
Seems way simpler indeed. I'll try to deploy the solution proposed in one device at least to verify everything. |
Beta Was this translation helpful? Give feedback.
In summary, I've successfully achieved a correct log rotation applying:
su root admto/etc/logrorate-kura.confin those devices based in Ubuntu server/opt/eclipse/kura/log4j/log4j.xmlaccording to #5962 and #5964. (specifically using the configuration on this file.I've tested the configuration in five different hardware and distributions (all of the Debian based, though) and all of them are working fine now. Other modifications proposed in the discussion did not succeed or they worked only partially.