High CPU consumption due to frequently Garbage Collect (GC) #31956

Syize · 2024-09-02T09:07:35Z

Description

Hi,

Recently I've been troubled by the issue of high CPU consumption of Gitea. The CPU consumption often become very high after running about 10 hours, and restart gitea can temporally solve this problem.

The most suspicious behavior of gitea during high CPU consumption is frequent GC. I have taken a screen record when cpu consumption becomes high.

gitea.mp4

After restarting Gitea, GC seems "calm down".

Here is the log of gitea (Though I can't find any related information.)

2024/09/02 05:06:14 ...s/setting/session.go:77:loadSessionFrom() [I] Session Service Enabled
2024/09/02 05:06:14 ...s/storage/storage.go:176:initAttachments() [I] Initialising Attachment storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/attachments
2024/09/02 05:06:14 ...s/storage/storage.go:166:initAvatars() [I] Initialising Avatar storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/avatars
2024/09/02 05:06:14 ...s/storage/storage.go:192:initRepoAvatars() [I] Initialising Repository Avatar storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/repo-avatars
2024/09/02 05:06:14 ...s/storage/storage.go:186:initLFS() [I] Initialising LFS storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/lfs
2024/09/02 05:06:14 ...s/storage/storage.go:198:initRepoArchives() [I] Initialising Repository Archive storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/repo-archive
2024/09/02 05:06:14 ...s/storage/storage.go:208:initPackages() [I] Initialising Packages storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/packages
2024/09/02 05:06:14 ...s/storage/storage.go:219:initActions() [I] Initialising Actions storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/actions_log
2024/09/02 05:06:14 ...s/storage/storage.go:223:initActions() [I] Initialising ActionsArtifacts storage with type: local
2024/09/02 05:06:14 ...les/storage/local.go:33:NewLocalStorage() [I] Creating new Local Storage at /var/lib/gitea/data/actions_artifacts
2024/09/02 05:06:14 routers/init.go:138:InitWebInstalled() [I] SQLite3 support is enabled
2024/09/02 05:06:14 routers/common/db.go:23:InitDBEngine() [I] Beginning ORM engine initialization.
2024/09/02 05:06:14 routers/common/db.go:30:InitDBEngine() [I] ORM engine initialization attempt #1/10...
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [I] PING DATABASE mysql
2024/09/02 05:06:14 ...dels/db/collation.go:184:preprocessDatabaseCollation() [W] Current database is using a case-insensitive collation "utf8mb4_unicode_ci", although Gitea could work with it, there might be some rare cases which don't work as expected.
2024/09/02 05:06:14 ...dels/db/collation.go:188:preprocessDatabaseCollation() [E] There are 278 table columns using inconsistent collation, they should use "utf8mb4_unicode_ci". Please go to admin panel Self Check page
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table system_setting Column version db default is , struct default is 1
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table gpg_key_import Column content db nullable is true, struct nullable is false
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project column board_type db type is INT(10) UNSIGNED, struct type is INT UNSIGNED
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project column card_type db type is INT(11), struct type is INT UNSIGNED
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project Column card_type db default is 0, struct default is 
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project Column card_type db nullable is false, struct nullable is true
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table project column type db type is INT(10) UNSIGNED, struct type is INT UNSIGNED
2024/09/02 05:06:14 cmd/web.go:194:serveInstalled() [W] Table review_state Column commit_sha db nullable is true, struct nullable is false
2024/09/02 05:06:15 cmd/web.go:194:serveInstalled() [W] Table label Column archived_unix db default is , struct default is NULL
2024/09/02 05:06:15 routers/init.go:144:InitWebInstalled() [I] ORM engine initialization successful!
2024/09/02 05:06:15 ...er/issues/indexer.go:76:func1() [I] PID 1132: Initializing Issue Indexer: bleve
2024/09/02 05:06:15 ...xer/stats/indexer.go:41:populateRepoIndexer() [I] Populating the repo stats indexer with existing repositories
2024/09/02 05:06:15 ...xer/stats/indexer.go:87:populateRepoIndexer() [I] Done (re)populating the repo stats indexer with existing repositories
2024/09/02 05:06:15 ...er/issues/indexer.go:153:func2() [I] Issue Indexer Initialization took 1.682015ms
2024/09/02 05:06:15 ...s/actions/cleanup.go:36:cleanExpiredArtifacts() [I] Found 0 expired artifacts
2024/09/02 05:06:15 ...s/actions/cleanup.go:60:cleanNeedDeleteArtifacts() [I] Found 0 artifacts pending deletion
2024/09/02 05:06:15 cmd/web.go:304:listen() [I] Listen: http://0.0.0.0:3000
2024/09/02 05:06:15 cmd/web.go:308:listen() [I] AppURL(ROOT_URL): http://gitea.syize.cn/
2024/09/02 05:06:15 cmd/web.go:311:listen() [I] LFS server enabled
2024/09/02 05:06:15 ...s/graceful/server.go:50:NewServer() [I] Starting new Web server: tcp:0.0.0.0:3000 on PID: 1132

Gitea Version

1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.46.0

Operating System

Arch Linux

How are you running Gitea?

I install the latest gitea from Arch Linux repo

pacman -Qi gitea

Name            : gitea
Version         : 1.22.1-1
Description     : Painless self-hosted Git service, community managed.
Architecture    : x86_64
URL             : https://gitea.io
Licenses        : MIT
Groups          : None
Provides        : None
Depends On      : git
Optional Deps   : mariadb: MariaDB support [installed]
                  memcached: MemCached support
                  openssh: GIT over SSH support [installed]
                  pam: Authentication via PAM support [installed]
                  postgresql: PostgreSQL support
                  redis: Redis support
                  sqlite: SQLite support [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 101.67 MiB
Packager        : Robin Candau <[email protected]>
Build Date      : 2024年08月04日 星期日 16时13分12秒
Install Date    : 2024年08月05日 星期一 11时16分13秒
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

and make it a system service by using systemd

sudo systemctl status gitea.service

● gitea.service - Gitea (Git with a cup of tea)
     Loaded: loaded (/usr/lib/systemd/system/gitea.service; enabled; preset: disabled)
     Active: active (running) since Mon 2024-09-02 16:14:55 CST; 51min ago
 Invocation: 0b2f0c623432438cbb02ae9595534496
   Main PID: 108749 (gitea)
      Tasks: 25 (limit: 38295)
     Memory: 173.8M (peak: 175.5M)
        CPU: 3.414s
     CGroup: /system.slice/gitea.service
             └─108749 /usr/bin/gitea web -c /etc/gitea/app.ini

Database

MySQL/MariaDB

The text was updated successfully, but these errors were encountered:

silverwind · 2024-09-02T10:30:30Z

Could be be the bleve indexer, see #31565 and try disabling it via REPO_INDEXER_ENABLED = false.

Syize · 2024-09-03T11:36:58Z

set REPO_INDEXER_ENABLED = false doesn't help.

Syize · 2024-09-07T12:27:00Z

set REPO_INDEXER_ENABLED = false doesn't help.

I set REPO_INDEXER_ENABLED = false and observed Gitea for a few days. This issue won't happen even Gitea has run 1-2 days, but it happens after system reboot, and the change of system time seems to be one of the causes triggering this problem.

Sometimes I switch to Windows to play games and then switch back to Arch Linux. Because of the different ways Windows and Linux handle the hardware clock (its a normal issue), the system time in Linux is 8 hours ahead(because I'm in UTC+8 timezone), and Linux will update system time as soon as it connect to Internet. I found this issue occurs immediately after system time update. Restart gitea manually could calm Gitea down.

As for Gitea's log, I have changed log level to trace, but Gitea doesn't output any additional information in log/gitea.log.

lunny · 2025-01-05T00:09:03Z

You need to change the log mode to file to record the log to file. Your logs uploaded don't record what happened when the GC is running.

Syize · 2025-01-05T11:58:42Z

The log mode had been MODE = console, file before I changed log level to Trace.

This issue hasn't occured again after I set REPO_INDEXER_ENABLED = false and resolved this time synchronization issue between Windows and Linux. It works fine now.

Sometimes I switch to Windows to play games and then switch back to Arch Linux. Because of the different ways Windows and Linux handle the hardware clock (its a normal issue), the system time in Linux is 8 hours ahead(because I'm in UTC+8 timezone), and Linux will update system time as soon as it connect to Internet. I found this issue occurs immediately after system time update. Restart gitea manually could calm Gitea down.

wxiaoguang · 2025-01-06T13:54:34Z

https://docs.gitea.com/help/support

If you encounter slow/hanging/deadlock problems, please report the stacktrace when the problem occurs. Go to the "Site Admin" -> "Monitoring" -> "Stacktrace" -> "Download diagnosis report".

coder-xiaomo · 2025-01-23T14:05:37Z

facing the same problem in gitea 1.23.1
CPU consumption is very high, and currently my REPO_INDEXER_ENABLED is set to false.
this problem cames after i update to version 1.23.1, older versions (1.22.x) didn't have this problem

Maybe there is another solution?

coder-xiaomo · 2025-01-23T14:09:07Z

the 内存分配次数 and 内存释放次数 number was growing very very fast, about 10,000 times in 1 second

wxiaoguang · 2025-01-24T00:40:20Z

https://docs.gitea.com/help/support

If you encounter slow/hanging/deadlock problems, please report the stacktrace when the problem occurs. Go to the "Site Admin" -> "Monitoring" -> "Stacktrace" -> "Download diagnosis report".

https://docs.gitea.com/help/support

If you encounter slow/hanging/deadlock problems, please report the stacktrace when the problem occurs. Go to the "Site Admin" -> "Monitoring" -> "Stacktrace" -> "Download diagnosis report".

wxiaoguang · 2025-01-24T00:42:35Z

And from your screenshot, I can see the the "free memory" is low. It seems that you are running a lot of services in a host which only has 3.4G memory. Maybe you could try to stop unrelated services or move Gitea to a new host to try.

wxiaoguang · 2025-01-24T00:46:09Z

In 1.23 , there is a license classifier, which consumes much more memory than 1.22 (100MB more) , so if the memory is low, the GC number could also become very high.

@yp05327

license classifier memory usage screenshot

yp05327 · 2025-01-24T01:58:35Z

The origin issue is in 1.22.1 and there's another issue in 1.23.1 which seems that it is caused by the new license detect feature.
About the issue in 1.22.1, I also met a simillar situation in 1.22.x once before, but can not reproduce it again.
About the issue in 1.23.1. If the memory usage of licenseclassifier should be considered as annoying, maybe we can add an option to disable this feature. IMO, It is similar to a indexer (or a search system) to compare between the detected license file and the sample files. I don't know how to reduce the mem usage from Gitea side, as it seems that these marked functions are all internal functions in licenseclassifier, and how to design a search system which uses very low mem usage and very fast search speed?
ps: Your instance which has no mem swap and less than 4GB mem is running Redis + Mysql+ Gitea + JAVA App + YDService and maybe more which seems too heavy.

silverwind · 2025-01-24T08:26:32Z

It is similar to a indexer

I don't see it similar. All that should be stored with licenseclassifier is is a simple map of repo to license, e.g. a small cached map, and that should consume miniscule amounts of memory.

While it is running, increased memory is acceptable, but if there is more in use after it has run, that would indicate unnecessary storage or a memory leak in the module.

yp05327 · 2025-01-24T09:13:05Z

licenseclassifier is used to classify/detect the provided license file belongs to which kind of license, not a map of repo to license (it is saved in DB)
The solution is loading the raw data of all known sample license files to licenseclassifier (in mem) when server start, then we can use it in any time instead of loading them every time. You can check the implementation in InitLicenseClassifier func.

All all known sample license files are in:
https://github.com/go-gitea/gitea/tree/main/options/license
and custom license files will also be loaded, IIRC.

The total size is 6.1M

silverwind · 2025-01-24T09:40:46Z

DB "cache" is fine too, but I'd advise against loading all license files into memory, that won't scale too well with instances that have millions of repos. Just re-write the DB entry when the repo is being pushed to.

silverwind · 2025-01-24T09:42:12Z

Maybe I misread. 6MB static sample licenses is fine of course.

lunny · 2025-01-24T19:37:02Z

DB "cache" is fine too, but I'd advise against loading all license files into memory, that won't scale too well with instances that have millions of repos. Just re-write the DB entry when the repo is being pushed to.

An advantage to putting those license information in a database is they can be updated on the fly but not always wait for another release.

wxiaoguang · 2025-01-31T09:02:21Z

Some of these discussions are off-topic. Let's focus on the problem:

If the GC consumes much CPU, it's highly likely that the memory is insufficient.
- Try to provide more memory for Gitea instance
- Help to get a diagnosis report from admin -> monitor -> trace panel to see what happens.
License detector consumes much more memory than before. By manually triggering the GC and watch memory usage:
- Without License Detector (skip its Init): Current Memory Usage 63 MiB
- With License Detector: Current Memory Usage 164 MiB
- Proposal: remove unpopular licenses Proposal: remove unpopular licenses #33467

Update:

The "license" memory usage should should have been fixed by Only keep popular licenses #33832 (in the current development branch "main" 1.24)

GiteaBot · 2025-03-03T20:30:58Z

We close issues that need feedback from the author if there were no new comments for a month. 🍵

wxiaoguang · 2025-03-10T16:05:23Z

The "license" memory usage should should have been fixed by Only keep popular licenses #33832 (in the current development branch "main" 1.24)

Syize added the type/bug label Sep 2, 2024

wxiaoguang mentioned this issue Jan 15, 2025

Support for OpenTelemetry #32866

Open

wxiaoguang added the issue/needs-feedback For bugs, we need more details. For features, the feature must be described in more detail label Feb 1, 2025

GiteaBot closed this as completed Mar 3, 2025

TheFox0x7 mentioned this issue Mar 5, 2025

Increased memory and CPU usage on 1.23.4 #33809

Closed

wxiaoguang mentioned this issue Mar 20, 2025

CPU 异常占用 #33926

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

High CPU consumption due to frequently Garbage Collect (GC) #31956

High CPU consumption due to frequently Garbage Collect (GC) #31956

Syize commented Sep 2, 2024 •

edited

Loading

silverwind commented Sep 2, 2024

Syize commented Sep 3, 2024

Syize commented Sep 7, 2024

lunny commented Jan 5, 2025

Syize commented Jan 5, 2025

wxiaoguang commented Jan 6, 2025

coder-xiaomo commented Jan 23, 2025

coder-xiaomo commented Jan 23, 2025 •

edited by wxiaoguang

Loading

wxiaoguang commented Jan 24, 2025

wxiaoguang commented Jan 24, 2025

wxiaoguang commented Jan 24, 2025 •

edited

Loading

yp05327 commented Jan 24, 2025

silverwind commented Jan 24, 2025 •

edited

Loading

yp05327 commented Jan 24, 2025 •

edited

Loading

silverwind commented Jan 24, 2025 •

edited

Loading

silverwind commented Jan 24, 2025

lunny commented Jan 24, 2025

wxiaoguang commented Jan 31, 2025 •

edited

Loading

GiteaBot commented Mar 3, 2025

wxiaoguang commented Mar 10, 2025

High CPU consumption due to frequently Garbage Collect (GC) #31956

High CPU consumption due to frequently Garbage Collect (GC) #31956

Comments

Syize commented Sep 2, 2024 • edited Loading

Description

Gitea Version

Can you reproduce the bug on the Gitea demo site?

Log Gist

Screenshots

Git Version

Operating System

How are you running Gitea?

Database

silverwind commented Sep 2, 2024

Syize commented Sep 3, 2024

Syize commented Sep 7, 2024

lunny commented Jan 5, 2025

Syize commented Jan 5, 2025

wxiaoguang commented Jan 6, 2025

coder-xiaomo commented Jan 23, 2025

coder-xiaomo commented Jan 23, 2025 • edited by wxiaoguang Loading

wxiaoguang commented Jan 24, 2025

wxiaoguang commented Jan 24, 2025

wxiaoguang commented Jan 24, 2025 • edited Loading

yp05327 commented Jan 24, 2025

silverwind commented Jan 24, 2025 • edited Loading

yp05327 commented Jan 24, 2025 • edited Loading

silverwind commented Jan 24, 2025 • edited Loading

silverwind commented Jan 24, 2025

lunny commented Jan 24, 2025

wxiaoguang commented Jan 31, 2025 • edited Loading

GiteaBot commented Mar 3, 2025

wxiaoguang commented Mar 10, 2025

Syize commented Sep 2, 2024 •

edited

Loading

coder-xiaomo commented Jan 23, 2025 •

edited by wxiaoguang

Loading

wxiaoguang commented Jan 24, 2025 •

edited

Loading

silverwind commented Jan 24, 2025 •

edited

Loading

yp05327 commented Jan 24, 2025 •

edited

Loading

silverwind commented Jan 24, 2025 •

edited

Loading

wxiaoguang commented Jan 31, 2025 •

edited

Loading