Add request/response body capture for network request tracking#933
Closed
Add request/response body capture for network request tracking#933
Conversation
Enhances network request tracking to capture and display request and response bodies, addressing a critical gap for agent workflows that need to analyze HTTP request/response content. ## Changes Made - **Extended TrackedRequest struct** to include: - `request_body`: Captures POST data from outgoing requests - `response_body`: Stores response content (fetched lazily after loading completes) - `status`: HTTP response status code - `response_headers`: Response headers from server - `request_id`: Internal CDP correlation ID - **Enhanced event handling** in `process_events()`: - `Network.requestWillBeSent`: Now extracts and stores request body (postData) - `Network.responseReceived`: Captures response metadata (status, headers) - `Network.loadingFinished`: Triggers lazy fetching of response body via `Network.getResponseBody` - **Added response body fetching** in `handle_requests()`: - Asynchronously fetches response bodies for completed requests - Handles both text and binary responses appropriately - Updates tracked requests with fetched content - **Updated CLI output formatting**: - Enhanced `format_request_table()` to display request/response bodies - Added `--clear` flag for clearing request logs - Improved command documentation ## Implementation Details - Response bodies are fetched lazily to avoid performance overhead during active browsing - Uses CDP's `Network.getResponseBody` API for reliable content retrieval - Maintains backward compatibility with existing network tracking features - Integrates seamlessly with existing HAR recording functionality This enables agents to inspect full HTTP request/response cycles including payloads, which is essential for understanding API interactions and web application behavior. Fixes #932
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Hi @ctate, I've been working on the same issue (#932) and opened #935 with an overlapping implementation. The core request/response capture is similar to yours, but my PR also adds filtering flags for
Happy to rebase on top of this PR if you'd prefer to merge yours first — I can then submit just the filtering additions as a follow-up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enhances network request tracking to capture and display request and response bodies, addressing a critical gap for agent workflows that need to analyze HTTP request/response content.
Changes Made
Extended TrackedRequest struct to include:
request_body: Captures POST data from outgoing requestsresponse_body: Stores response content (fetched lazily after loading completes)status: HTTP response status coderesponse_headers: Response headers from serverrequest_id: Internal CDP correlation IDEnhanced event handling in
process_events():Network.requestWillBeSent: Now extracts and stores request body (postData)Network.responseReceived: Captures response metadata (status, headers)Network.loadingFinished: Triggers lazy fetching of response body viaNetwork.getResponseBodyAdded response body fetching in
handle_requests():Updated CLI output formatting:
format_request_table()to display request/response bodies--clearflag for clearing request logsImplementation Details
Network.getResponseBodyAPI for reliable content retrievalThis enables agents to inspect full HTTP request/response cycles including payloads, which is essential for understanding API interactions and web application behavior.
Fixes #932