Memory leak

[ljlooper]

At first, a connection was created, and 100,000 pieces of data were inserted, followed by a random id query every 20 seconds, after the query, the rows were closed, and there was no query blocking, I confirmed that the code was OK, but the memory continued to grow from the initial 200M, and now has reached more than 1g, and continues to increase

[jlmadurga]

I'm having also issues with memory. In my case I'm using Session and it seems there is a leak altough I'm closing it.

I'm case using:

• session :memory:
• chdb 3.0.0

Check this script to reproduce it with memray, I'm using a file with 3MB of JSON data.

import memray
import chdb
from chdb import session
import os
import time

def load_data(file_path):
    """Simulates loading a file into memory as a string."""
    with open(file_path, "r") as f:
        return f.read()

def chdb_memory_test(data):
    """Creates a chDB session, inserts data, and closes the session."""
    with session.Session(":memory:") as ses:
        ses.query("CREATE TABLE test (data String) Engine = Memory")
        ses.query(f"INSERT INTO test FORMAT JSONEachRow {data}")


if __name__ == "__main__":
    file_path = "events.json"

    data = load_data(file_path)

    with memray.Tracker("memray_report.bin"):
        for _ in range(50):  # Run multiple iterations to confirm leaks
            chdb_memory_test(data)
            time.sleep(0.1)  # Simulate processing time

    print("Memray tracking complete. Analyze with:")
    print("memray flamegraph memray_report.bin")

Executing it there is a warning:
UserWarning: There is already an active session. Creating a new session will close the existing one. It is recommended to close the existing session before creating a new one. Closing the existing session :memory:

but I guess this is caused by not setting to None g_session on close() https://github.com/chdb-io/chdb/blob/main/chdb/session/state.py#L86

[auxten]

Thanks for reporting @jlmadurga @ljlooper. Let me check it.

[auxten]

@ljlooper are you also using the memory engine which is the default engine of Session and Connection

[ljlooper]

yes, Please review my test code
`
func main() {

    db, err := sql.Open("chdb", "session=/opt/data/chdb_test_parquet;driverType=Parquet")
    if err != nil {
        fmt.Println("open:", err)
        return
    }
    defer db.Close()

    fmt.Println("open db success")
    _, err = db.Exec("create database if not exists tdb")
    if err != nil {
        fmt.Println("create database ", err)
        return
    }
    _, err = db.Exec(`CREATE TABLE IF NOT EXISTS tdb.test
            (
                id String,
                etag String,					  
                contents String
            )ENGINE = MergeTree()
            PRIMARY KEY (id)
            ORDER BY (id);`)
    if err != nil {
        fmt.Println("create table ", err)
        return
    }

    batchWrite(db)

    go func() {
        http.ListenAndServe("0.0.0.0:12346", nil)
    }()

    tk := time.NewTicker(20 * time.Second)
    for {
        <-tk.C
        // query(db)
        ConcurrencyQuery(10, db)
    }
}

func ConcurrencyQuery(n int, db *sql.DB) {

    var wg sync.WaitGroup
    for i := 0; i < n; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            query(db)
        }()
    }

    wg.Wait()
}

func batchWrite(db *sql.DB) {
    t0 := time.Now()
    count := 100000
    sqlBuf := bytes.NewBufferString("INSERT INTO tdb.test VALUES")
    for i := 1; i <= count; i++ {
        if i%1000 == 0 {
            sqlBuf.WriteString(fmt.Sprintf(` ('%d', 'xx','xx')`, i%1000))
            _, err := db.Exec(sqlBuf.String())
            if err != nil {
                panic("write: " + err.Error())
            }
            sqlBuf = bytes.NewBufferString("INSERT INTO tdb.test VALUES")
        }
        sqlBuf.WriteString(fmt.Sprintf(` ('%d', 'xx', 'xx'),`, i%1000))
    }
    fmt.Printf("write count: %d, used: %+v\n", count, time.Since(t0))
}

func query(db *sql.DB) {
    t0 := time.Now()
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    row, err := db.QueryContext(ctx, fmt.Sprintf(`select count(*) from tdb.test where id='%d'`, rand.Intn(100)))
    if err != nil {
        panic("query:" + err.Error())
    }
    defer row.Close()

    var count = 0
    for row.Next() {
        err := row.Scan(&count)
        if err != nil {
            panic("scan: " + err.Error())
        }
    }
    fmt.Println("count:", count)
    fmt.Printf("qeury used: %+v\n", time.Since(t0))
}

`

[auxten]

@ljlooper Oh, You are using the chdb-go which is still using the libchdb 2.x.
I think you issue and @jlmadurga should be 2 different issues.
Any way, I will look into them.

[catundercar]

@ljlooper Oh, You are using the chdb-go which is still using the libchdb 2.x. I think you issue and @jlmadurga should be 2 different issues. Any way, I will look into them.

After preliminary investigation, I used Session and go SQL Driver, and tested the two driver types of ARROW and PARQUET, both of which have memory leakage problems. After the above code reduces the ticker interval, the phenomenon will be more obvious.

[auxten]

Ok, there are mainly 2 problems here:

1. The clickhouse engine continue eating memory even idle, in my gears it will hold about 1.9G memory. We need to find a way to let user take control on it.
2. The second issue is a real memory leak caused by a typo which is fixed here.

Currently, problem 2 is fixed. Continue to address problem 1.

[auxten]

BTW, most memory alloc and free are happened in C++ code. So, we cannot use memray to address these problems. I usually take valgrind here. Here is the output after #299

data = f'{{"data": "{"a"*1000,000}"}}'
# ses = session.Session("file::memory:?verbose&log-level=test")
ses = session.Session(":memory:")
ses.query("CREATE TABLE test (data String) Engine = Memory")
for _ in range(100):
    ses.query(f"INSERT INTO test FORMAT JSONEachRow {data}")
for i in range(100):
    ret = ses.query("SELECT * FROM test")
    if i == 0:
        print("First query result:", len(ret))
ses.close()

gc.collect()

time.sleep(3)

==1489041== LEAK SUMMARY:
==1489041==    definitely lost: 0 bytes in 0 blocks
==1489041==    indirectly lost: 0 bytes in 0 blocks
==1489041==      possibly lost: 5,032 bytes in 9 blocks
==1489041==    still reachable: 507,665 bytes in 271 blocks
==1489041==         suppressed: 0 bytes in 0 blocks

[catundercar]

It may not be caused by memory leakage, and it may be controlled by the parameter max_server_memory_usage.

[auxten]

I belive it's fixed in #304.
chDB v3.0.1 is roasting