Errors in clickhouse shard, about memory, whatever the allocated memory

[stouch]

Since recently, I got every few seconds this error :

│ 2025.02.16 18:51:39.912044 [ 625 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {156b9c34-a399-4572-b3c9-2a724652dae3::202502_108261_112680_236}: C │
│ ode: 241. DB::Exception: Memory limit (total) exceeded: would use 4.40 GiB (attempt to allocate chunk of 4362064 bytes), current RSS 1.22 GiB, maximum: 4.39 GiB. OvercommitTracker deci │
│ sion: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines be │
│ low):                                                                                                                                                                                    │
│                                                                                                                                                                                          │
│ 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d66ad9b                                                                                              │
│ 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007e75b2c                                                                                                             │
│ 2. DB::Exception::Exception<char const*, char const*, String, long&, String, String, char const*, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std: │
│ :type_identity<char const*>::type, std::type_identity<char const*>::type, std::type_identity<String>::type, std::type_identity<long&>::type, std::type_identity<String>::type, std::type │
│ _identity<String>::type, std::type_identity<char const*>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, char const*&&, char const*&&, String&&, │
│  long&, String&&, String&&, char const*&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000000d689064                                                                  │
│ 3. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d68818d                                                                                                     │
│ 4. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d687cb1                                                                                                     │
│ 5. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d687cb1                                                                                                     │
│ 6. MemoryTracker::allocImpl(long, bool, MemoryTracker*, double) @ 0x000000000d687cb1                                                                                                     │
│ 7. Allocator<false, false>::alloc(unsigned long, unsigned long) @ 0x000000000d63c1b0                                                                                                     │
│ 8. DB::Memory<Allocator<false, false>>::alloc(unsigned long) @ 0x000000000d6ce995                                                                                                        │
│ 9. DB::WriteBufferFromFileDescriptor::WriteBufferFromFileDescriptor(int, unsigned long, char*, std::shared_ptr<DB::Throttler>, unsigned long, String, bool, unsigned long) @ 0x000000000 │
│ d6f4ac1                                                                                                                                                                                  │
│ 10. DB::WriteBufferFromFile::WriteBufferFromFile(String const&, unsigned long, int, std::shared_ptr<DB::Throttler>, unsigned int, char*, unsigned long, bool, unsigned long) @ 0x0000000 │
│ 00d815b4f                                                                                                                                                                                │
│ 11. DB::DiskLocal::writeFile(String const&, unsigned long, DB::WriteMode, DB::WriteSettings const&) @ 0x00000000111156f3                                                                 │
│ 12. DB::DataPartStorageOnDiskFull::writeFile(String const&, unsigned long, DB::WriteMode, DB::WriteSettings const&) @ 0x0000000012992953                                                 │
│ 13. DB::MergeTreeDataPartWriterOnDisk::Stream<false>::Stream(String const&, std::shared_ptr<DB::IDataPartStorage> const&, String const&, String const&, String const&, String const&, st │
│ d::shared_ptr<DB::ICompressionCodec> const&, unsigned long, std::shared_ptr<DB::ICompressionCodec> const&, unsigned long, DB::WriteSettings const&) @ 0x0000000012b5caf6                 │
│ 14. void std::__function::__policy_invoker<void (DB::ISerialization::SubstreamPath const&)>::__call_impl<std::__function::__default_alloc_func<DB::MergeTreeDataPartWriterWide::addStrea │
│ ms(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::shared_ptr<DB::IAST> const&)::$_0, void (DB::ISerialization::SubstreamPath const&)>>(std::__fun │
│ ction::__policy_storage const*, DB::ISerialization::SubstreamPath const&) @ 0x0000000012b6c4f7                                                                                           │
│ 15. DB::SerializationSparse::enumerateStreams(DB::ISerialization::EnumerateStreamsSettings&, std::function<void (DB::ISerialization::SubstreamPath const&)> const&, DB::ISerialization:: │
│ SubstreamData const&) const @ 0x0000000010f2862a                                                                                                                                         │
│ 16. DB::ISerialization::enumerateStreams(std::function<void (DB::ISerialization::SubstreamPath const&)> const&, std::shared_ptr<DB::IDataType const> const&, COW<DB::IColumn>::immutable │
│ _ptr<DB::IColumn> const&) const @ 0x0000000010ebbc88                                                                                                                                     │
│ 17. DB::MergeTreeDataPartWriterWide::addStreams(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::shared_ptr<DB::IAST> const&) @ 0x0000000012b6445f  │
│ 18. DB::MergeTreeDataPartWriterWide::MergeTreeDataPartWriterWide(String const&, String const&, std::unordered_map<String, std::shared_ptr<DB::ISerialization const>, std::hash<String>,  │
│ std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization const>>>> const&, std::shared_ptr<DB::IDataPartStorage>, DB::MergeTreeIndexGranularityI │
│ nfo const&, std::shared_ptr<DB::MergeTreeSettings const> const&, DB::NamesAndTypesList const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::VirtualCol │
│ umnsDescription const> const&, std::vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<D │
│ B::ColumnStatistics>, std::allocator<std::shared_ptr<DB::ColumnStatistics>>> const&, String const&, std::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, D │
│ B::MergeTreeIndexGranularity const&) @ 0x0000000012b6425f                                                                                                                                │
│ 19. DB::createMergeTreeDataPartWriter(DB::MergeTreeDataPartType, String const&, String const&, std::unordered_map<String, std::shared_ptr<DB::ISerialization const>, std::hash<String>,  │
│ std::equal_to<String>, std::allocator<std::pair<String const, std::shared_ptr<DB::ISerialization const>>>> const&, std::shared_ptr<DB::IDataPartStorage>, DB::MergeTreeIndexGranularityI │
│ nfo const&, std::shared_ptr<DB::MergeTreeSettings const> const&, DB::NamesAndTypesList const&, std::unordered_map<String, unsigned long, std::hash<String>, std::equal_to<String>, std:: │
│ allocator<std::pair<String const, unsigned long>>> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::shared_ptr<DB::VirtualColumnsDescription const> const&, std:: │
│ vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<DB::ColumnStatistics>, std::allocator │
│ <std::shared_ptr<DB::ColumnStatistics>>> const&, String const&, std::shared_ptr<DB::ICompressionCodec> const&, DB::MergeTreeWriterSettings const&, DB::MergeTreeIndexGranularity const&) │
│  @ 0x0000000012b5a67b                                                                                                                                                                    │
│ 20. DB::MergedBlockOutputStream::MergedBlockOutputStream(std::shared_ptr<DB::IMergeTreeDataPart> const&, std::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::NamesAndTypesLis │
│ t const&, std::vector<std::shared_ptr<DB::IMergeTreeIndex const>, std::allocator<std::shared_ptr<DB::IMergeTreeIndex const>>> const&, std::vector<std::shared_ptr<DB::ColumnStatistics>, │
│  std::allocator<std::shared_ptr<DB::ColumnStatistics>>> const&, std::shared_ptr<DB::ICompressionCodec>, DB::TransactionID, bool, bool, DB::WriteSettings const&, DB::MergeTreeIndexGranu │
│ larity const&) @ 0x0000000012cada3b                                                                                                                                                      │
│ 21. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() const @ 0x0000000012a4060c                                                                                                │
│ 22. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x0000000012a472ce                                                                                                      │
│ 23. DB::MergeTask::execute() @ 0x0000000012a4ff22                                                                                                                                        │
│ 24. DB::MergePlainMergeTreeTask::executeStep() @ 0x0000000012e46398                                                                                                                      │
│ 25. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x0000000012a6f99d                                                                                      │
│ 26. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false, true>, void*>) @ 0x000000000d740b38                               │
│ 27. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void ThreadPoolIm │
│ pl<ThreadFromGlobalPoolImpl<false, true>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__ │
│ function::__policy_storage const*) @ 0x000000000d745c71                                                                                                                                  │
│ 28. void* std::__thread_proxy[abi:v15007]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<vo │
│ id>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000d74409f                                                                     │
│ 29. ? @ 0x00007f5282389144    

So I tried to increase the memory of clickhouse in the config, but whatever the memory is, even 10Gi, I finally got this (with maximum: <WHATEVER_MAX_MEMORY_THAT_I_SET> GiB in the above error message).

Any idea ?

Thanks

[stouch]

These

<Error> MergeTreeBackgroundExecutor: Exception while executing background task {156b9c34-a399-4572-b3c9-2a724652dae3::202502_108261_121167_236}: Code: 241. DB::Exception:  │
│ Memory limit (total) exceeded: would use 4.39 GiB (attempt to allocate chunk of 4361728 bytes), current RSS 1.07 GiB, maximum: 4.39 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time o │
│ r overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

errors still intensify...

[echozio]

ClickHouse needs a lot of memory. I don't know what specifically is causing your issue, but some of the recommendations for running on low memory might help you out: https://clickhouse.com/docs/en/operations/tips#using-less-than-16gb-of-ram

[stouch]

Could we have this documented in the chart clickhouse config part ? It's not obvious to get at least 32GB + all the ram for the rest of services , to just collect and transmit events.

Btw, the way my questions are answered (about default clickhouse system tables size, and ingest pod persistence, and kafka replications issues, and now about clickhouse memory) makes me think Jitsu really use by default a quantity of memory and services quite insane.. for the usage we have in dispatching about hundred of thousands events per day.

And the new docs is so light .. https://docs.jitsu.com/,

Anyway, I 'd really like to get more documentation in jitsu-charts about all this.

[echozio]

Managing these services, especially Jitsu's third party dependencies, over time is not really within the scope of this chart. We recommend managing them separately for a production deployment. I'm not opposed to including some useful documentation on how to manage them, but the best source for this is likely going to be the official documentation for each service. We're operating a relatively small instance of Jitsu, so we don't necessarily have all the answers when it comes to scaling beyond that, but we're open to contributions from those who are processing a greater number of events.