Skip to content
This repository was archived by the owner on Jan 24, 2025. It is now read-only.

Add cache to avoid loading committee from KVStore for every block #713

Closed
piotrm50 opened this issue Feb 1, 2024 · 1 comment · Fixed by #814
Closed

Add cache to avoid loading committee from KVStore for every block #713

piotrm50 opened this issue Feb 1, 2024 · 1 comment · Fixed by #814
Assignees
Labels
bug Something isn't working team-consensus Issues for Consensus Team
Milestone

Comments

@piotrm50
Copy link
Collaborator

piotrm50 commented Feb 1, 2024

          This is a hot path that reads data from the committee KVStore for each block - we should add a cache at some level. Maybe we should have a committee cache in SeatManager and clear it whenever SybilProtection.Reset() is called? Not necessarily as part of this PR, but then we should create an issue for that.

Originally posted by @piotrm50 in #673 (comment)

Might also be an issue in other places

@piotrm50 piotrm50 converted this from a draft issue Feb 1, 2024
@piotrm50 piotrm50 added nice-to-have team-consensus Issues for Consensus Team labels Feb 1, 2024
@piotrm50 piotrm50 added this to the v1.0.0-beta milestone Feb 1, 2024
@piotrm50 piotrm50 moved this from Product Backlog to Backlog in iota-core Mar 4, 2024
@jonastheis
Copy link
Contributor

The extend of this can be easily seen when running a test with a big committee: almost 2/3 of the entire runtime is spent loading the committee.

image

Steps to reproduce:

go test -run=TestBigCommittee -tags rocksdb,dockertests -v -failfast=true -count=1 -cpuprofile big-committee.profile.out
go tool pprof -http=:8181 big-committee.profile.out

file: big_commitee_test.go

package tests

import (
	"fmt"
	"testing"

	"github.com/samber/lo"

	"github.com/iotaledger/iota-core/pkg/testsuite"
	iotago "github.com/iotaledger/iota.go/v4"
)

func TestBigCommittee(t *testing.T) {
	var (
		genesisSlot       iotago.SlotIndex = 0
		minCommittableAge iotago.SlotIndex = 2
		maxCommittableAge iotago.SlotIndex = 4
	)

	ts := testsuite.NewTestSuite(t,
		testsuite.WithProtocolParametersOptions(
			iotago.WithTimeProviderOptions(
				genesisSlot,
				testsuite.GenesisTimeWithOffsetBySlots(1000, testsuite.DefaultSlotDurationInSeconds),
				testsuite.DefaultSlotDurationInSeconds,
				3,
			),
			iotago.WithLivenessOptions(
				10,
				10,
				minCommittableAge,
				maxCommittableAge,
				5,
			),
		),
	)
	// defer ts.Shutdown()

	for i := 0; i < 32; i++ {
		nodeName := fmt.Sprintf("node%d", i)
		ts.AddValidatorNode(nodeName)
	}

	for i := 32; i < 50; i++ {
		nodeName := fmt.Sprintf("node%d", i)
		ts.AddNode(nodeName)
	}

	ts.Run(true)
	fmt.Println("TestBigCommittee")

	ts.IssueBlocksAtSlots("", lo.RangeFrom(iotago.SlotIndex(1), 5), 4, "Genesis", ts.Nodes(), true, false)

	ts.AssertNodeState(ts.Nodes(),
		testsuite.WithLatestCommitmentSlotIndex(3),
		testsuite.WithEqualStoredCommitmentAtIndex(3),
	)
}

@piotrm50 piotrm50 added the bug Something isn't working label Mar 4, 2024
@jonastheis jonastheis moved this from Backlog to In Progress in iota-core Mar 7, 2024
@jonastheis jonastheis moved this from In Progress to In Review in iota-core Mar 7, 2024
@github-project-automation github-project-automation bot moved this from In Review to Done in iota-core Mar 7, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working team-consensus Issues for Consensus Team
Projects
Archived in project
2 participants