Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Organize chronicle exporter by http and grpc #2044

Open
wants to merge 10 commits into
base: release/v1.68.0
Choose a base branch
from

Conversation

djaglowski
Copy link
Member

This PR contains many changes so it may be wise to chunk it out into smaller changes. However, I'm opening it as a draft in order to facilitate feedback on the overall direction.

Changes included:

  • Separate "http exporter" and "grpc exporter".
  • Host metrics collection code is more clearly limited to the grpc exporter.
  • Separate marshaling code & deletes mocking of marshaler. We shouldn't be mocking things that are end-to-end testable in a local environment.
  • Implement mock servers to maximize code coverage. Specifically, this allows us to exercise most of the client code.
  • It's still necessary to mock a few functions, basically limited to building endpoints and bypassing authentication.

Future work:

  • No changes to configuration for the sake of backwards compatibility. However, it's quite clear that having all configuration at the same level in a single struct makes it difficult to logically enforce the rules. We can consider whether to migrate this later.
  • This does not yet include tests for splitting payloads, but we should be able to add them more easily now, since we are not mocking the entire clients.

exporter/chronicleexporter/util_test.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/marshal.go Outdated Show resolved Hide resolved
exporter/chronicleexporter/http_exporter.go Outdated Show resolved Hide resolved
@djaglowski djaglowski marked this pull request as ready for review December 17, 2024 18:58
@djaglowski
Copy link
Member Author

Several smaller PRs have been peeled off and merged. What's left is a large diff but is almost entirely just moving the marshaler into an internal package and separating the functionality into http and grpc variants. The benefit of doing this is that it is now tested according to its exported surface area only, but without exposing the exported surface area to other packages.

djaglowski and others added 4 commits December 18, 2024 10:21
* fix: Shut down zombie goroutine in chronicleexporter (#2029)

* Properly shut down chronicleexporter zombie goroutine

* Fix lint

* Fix the same problem for the GRPC workflow

* initial structure & getting headers

* progress

* custom messages sending & receiving properly

* more progress

* more changes

* cleanup

* dont use TopologyInterval, TODO: remove TopologyInterval from BP Extension

* cleanup

* cleanup & tests

* rm print statements

* cleanup, tests, fix tests

* fix bp extension logic, fix lint

* add gatewayid parameter

* fix concurrent map write

* fix tests

* fix test

* cleanup names & data model, add ResourceNameHeader

* fix resource name header

* address pr feedback

* fix lint

* repo rename fixes

* fix rebase issue

* fix gomod versions

* fix go mod

* update topo proc go mod

* fix flaky test

* bump version to 1.68.0

---------

Co-authored-by: Ian Adams <[email protected]>
@djaglowski djaglowski changed the base branch from release/v1.67.0 to release/v1.68.0 December 20, 2024 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants