Skip to content

Commit 76c3a73

Browse files
docs: eBPF HTTPS capture docs, runbooks, kind e2e scripts (AC-1059)
- Update CLAUDE.md: Go 1.25, ebpf/ package layout, charts/ location, kube webhook - Add docs/webhook-runbook.md: production webhook deployment runbook - Add docs/kind-e2e-demo-presentation.md: kind cluster demo walkthrough - Add test/kind/SERVICE-TEST-GUIDE.md: guide for testing all language workloads - Add test/kind/deploy-e2e-demo.sh + deploy-e2e-prod.sh: end-to-end deploy scripts - Update test/kind/workloads.yaml: consolidated workload manifest Note: docs/phases/ (internal dev notes) intentionally excluded
1 parent 0c70eac commit 76c3a73

8 files changed

Lines changed: 2262 additions & 0 deletions

File tree

CLAUDE.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# CLAUDE.md
2+
3+
Guidance for Claude Code (and other AI coding agents) when working in this repository.
4+
5+
## Project overview
6+
7+
This is the **Postman Insights Agent** — an open-source Go CLI that captures HTTP traffic from network interfaces (via libpcap), parses/witnesses requests and responses, and ships them to the Postman Insights backend so users can see endpoints, errors, latency, and volume in Postman.
8+
9+
- **Language:** Go (see `go.mod``go 1.25.0`, toolchain `go1.25.11`).
10+
- **Module:** `github.com/postmanlabs/postman-insights-agent`
11+
- **Entry point:** `main.go``cmd.Execute()` (Cobra-based CLI).
12+
- **Binary name:** `postman-insights-agent`
13+
- **License:** See `LICENSE`.
14+
- **Origin:** Forked from / successor to the Akita Software agent — many internal packages still reference `akita*` and `akitasoftware/*` libs.
15+
16+
Note: This open-source repo does **not** include type/format inference. That functionality only exists in the official distributed binary.
17+
18+
## Build, test, run
19+
20+
System prerequisites:
21+
- Go 1.25 (repo pins `go1.25.0` / toolchain `go1.25.11`).
22+
- `libpcap`
23+
- macOS: `brew install libpcap`
24+
- Debian/Ubuntu: `apt-get install libpcap-dev`
25+
- `mockgen` for tests: `go install github.com/golang/mock/mockgen@v1.5.0`
26+
- For eBPF features: `clang`, `llvm`, `linux-headers` (CI installs these automatically; see `.circleci/config.yml`).
27+
28+
Common commands (from `Makefile`):
29+
30+
| Command | What it does |
31+
| --------------- | ------------ |
32+
| `make` | `make build` — produces `bin/postman-insights-agent` |
33+
| `make build` | `go build -o bin/postman-insights-agent .` (runs `clean` first) |
34+
| `make clean` | `go clean` |
35+
| `make mock` | `go generate ./rest` (regenerates gomock mocks) |
36+
| `make test` | `make mock` then `go test ./...` |
37+
| `make docker-build` | Build via `build-scripts/Dockerfile`, output binary to `bin/` |
38+
39+
CI (`.circleci/config.yml`) runs `make` and `gotestsum --junitfile ...` against all packages.
40+
41+
## Repository layout
42+
43+
Top-level CLI plumbing:
44+
- `main.go` — trivial entry point.
45+
- `cmd/` — Cobra command tree (`root.go`, `supervisor.go`).
46+
- `cmd/internal/` — subcommand implementations:
47+
- `apidump/``apidump` command: the core traffic-capture command.
48+
- `apidump-ebpf/` — hidden `apidump-ebpf` command: credential-free eBPF capture for local dev/debugging.
49+
- `ec2/``ec2 setup|remove`: install agent as a systemd service on EC2.
50+
- `ecs/``ecs add|remove|cf-fragment|task-def`: AWS ECS integration.
51+
- `kube/``kube inject|run|secret|helm-fragment|tf-fragment|webhook`: Kubernetes integration (DaemonSet, sidecar injection, manifests, mutating webhook).
52+
- `kube-webhook/``kube webhook` subcommand: HTTPS admission webhook server that auto-injects the Java agent into pods.
53+
- `legacy/` — legacy `specs` and other deprecated commands.
54+
- `ascii/`, `akiflag/`, `cmderr/`, `pluginloader/` — CLI utilities.
55+
56+
Core domain packages:
57+
- `apidump/` — orchestrates a capture session: pcap → parse → trace → backend.
58+
- `pcap/` — libpcap wrappers, packet/stream reassembly, replay.
59+
- `learn/` — HTTP parsing (request/response → IR "witnesses"), JSON preprocessing, event-stream parsing, Luhn checks.
60+
- `trace/` — collectors (`backend_collector`, `dummy_collector`), rate limiting, stats, filters, reporting buffer.
61+
- `rest/` — HTTP clients for the Postman/Akita backend (`front_client`, `learn_client`, base client, auth, errors). Mocks are generated here via `go generate`.
62+
- `daemon/` — long-running daemon HTTP server.
63+
- `plugin/` — pluggable architecture (`interface.go`, `akita/` plugin).
64+
- `ebpf/` — eBPF-based HTTPS capture (libssl uprobes + Java TLS kprobe). Key sub-packages:
65+
- `ebpf/loader/` — loads BPF programs via `bpf2go`; generated `*_bpfel.go` files are gitignored (regenerated by CI).
66+
- `ebpf/events/` — Go-side event types, ring-buffer reader, HTTP/2 adapter.
67+
- `ebpf/programs/` — BPF C source (`libssl.bpf.c`, `java_tls.bpf.c`, `event.h`).
68+
- `ebpf/discovery/` — /proc-based PID watcher for auto-discovery of libssl processes.
69+
70+
Integrations and platform helpers:
71+
- `integrations/cri_apis/` — Container Runtime Interface client (containerd/CRI-O).
72+
- `integrations/kube_apis/` — Kubernetes API access.
73+
- `integrations/nginx/` — nginx-related helpers.
74+
- `integrations/tests/` — integration tests.
75+
- `aws_utils/` — AWS SDK helpers.
76+
- `tcp_conn_tracker/`, `tls_conn_tracker/` — connection tracking.
77+
- `useragent/`, `location/`, `version/`, `setversion/`, `telemetry/`, `usage/`, `printer/`, `consts/`, `cfg/`, `env/`, `util/` — utilities.
78+
79+
Other:
80+
- `data_masks/` — PII/sensitive data redaction.
81+
- `apispec/` — API spec generation/handling.
82+
- `architecture/architecture.go` — architecture metadata.
83+
- `docs/discovery-mode.md` — important user-facing doc explaining **Discovery Mode** vs **Workspace Mode** onboarding (Kubernetes). Read this before changing onboarding/k8s flows.
84+
- `charts/postman-insights-webhook/` — Helm chart for the mutating admission webhook (Java agent auto-injection). This is the canonical chart location; do not put charts under `deployment/`.
85+
- `deployment/` — Go package (`package deployment`) for deployment-environment detection (EC2, ECS, k8s, etc.). **Not** for chart artifacts.
86+
- `build-scripts/`, `ci/` — release & deployment scripts. `build-scripts/Dockerfile` uses `golang:1.25-alpine` for static binary builds (Alpine's libpcap has no D-Bus dependency).
87+
88+
## CLI surface (high level)
89+
90+
Root: `postman-insights-agent` (see `cmd/root.go`). Notable subcommands:
91+
92+
- `apidump` — Capture requests/responses from network traffic (primary command). Supports `--enable-https-capture` (eBPF libssl) and `--enable-java-tls` (Java TLS kprobe).
93+
- `ec2 setup` / `ec2 remove` — Install/uninstall as systemd service on EC2.
94+
- `ecs add|remove|cf-fragment|task-def` — Manage agent on AWS ECS.
95+
- `kube inject|run|secret|helm-fragment|tf-fragment` — Kubernetes integration. `kube run` is the DaemonSet entrypoint (Linux only).
96+
- `kube webhook` — Run the mutating admission webhook that auto-injects the Postman Java agent into pods in opted-in namespaces. Runs as a separate Deployment in the cluster (not the DaemonSet); deployed via `charts/postman-insights-webhook/`.
97+
- `specs` (legacy) — Manage API specs.
98+
- `aki` — ASCII art easter egg.
99+
100+
Test-only / hidden flags live on the root command (`testOnlyUseHTTPSFlag`, `dogfoodFlag`, `debugFlag`, profiling flags, etc.) — see `cmd/root.go`.
101+
102+
## Conventions and gotchas
103+
104+
- **Cobra + Viper + pflag** are used throughout. New subcommands should follow the pattern in `cmd/internal/<name>/<name>.go` and be wired into `cmd/root.go`.
105+
- **Mocks**: `rest/` uses gomock. Run `make mock` (or `go generate ./rest`) after changing interfaces in `rest/interface.go`. `make test` does this automatically.
106+
- **libpcap is required at build time** on every platform — `pcap/get_pcap_handle_linux.go` is Linux-only; the generic file covers other OSes.
107+
- **Linux-only features**: `kube run` (DaemonSet mode), eBPF capture, and some pcap paths assume Linux. Be mindful when adding code under build-tagged files.
108+
- **eBPF generated files**: `ebpf/loader/*_bpfel.go` and `*.o` are produced by `go generate ./ebpf/loader` (via `bpf2go`) and are gitignored — CI regenerates them from source. Do not commit them.
109+
- **Helm charts** live in `charts/`, not `deployment/`. The `deployment/` directory is a Go package for environment detection.
110+
- **Akita lineage**: Many package names, struct names, and dependencies still say `akita`. Do **not** rename these casually — they cross module boundaries (`github.com/akitasoftware/akita-ir`, `akita-libs`, `go-utils`). When in doubt, leave the name alone.
111+
- **Backend talk** goes through `rest/` clients. Avoid hitting external HTTP from other packages directly.
112+
- **Telemetry** is in `telemetry/`; usage events in `usage/`. Keep new event names consistent with existing ones.
113+
- **Discovery vs Workspace mode**: see `docs/discovery-mode.md`. New k8s onboarding work should support both and avoid resurrecting the legacy `--project` / `--collection` flags.
114+
- **Code owners**: see `.github/CODEOWNERS` before opening PRs.
115+
- **Versioning**: SemVer; bump via files under `version/` and `setversion/` as documented in `CONTRIBUTING.md`.
116+
117+
## Running locally
118+
119+
```bash
120+
make # builds bin/postman-insights-agent
121+
./bin/postman-insights-agent --help # explore commands
122+
./bin/postman-insights-agent apidump --help
123+
```
124+
125+
Capturing traffic typically requires root / `CAP_NET_RAW` (libpcap). Use `sudo` locally or run inside the provided container.
126+
127+
## When making changes
128+
129+
1. Prefer small, well-scoped edits. The codebase mixes legacy Akita code with newer Postman code — don't refactor broadly without reason.
130+
2. If you touch anything in `rest/`, regenerate mocks (`make mock`) and run `make test`.
131+
3. If you add a subcommand, add it under `cmd/internal/<name>/` and register it in `cmd/root.go`.
132+
4. If you change onboarding/k8s behavior, update `docs/discovery-mode.md`.
133+
5. Update `README.md` per `CONTRIBUTING.md` when changing user-visible interface (env vars, flags, ports).
134+
6. Run `make test` before claiming work is done. CI is `make` + `make mock` + `gotestsum`.

0 commit comments

Comments
 (0)