Raw Request / Response content (and other feature suggestions)

[TrapperHell]

I wanted to first thank you for this awesome-looking app. I am a long-time Fiddler user, but I have been looking for some more modern and unencumbered alternatives and this really stood out.

What I'm really missing at the moment is a raw text display of the complete request (headers + body) and complete response content (headers + body) that is easily copyable. (You can think of the current text-display of the request / response content but applicable to the headers as well). Along with that I think it would be nice to contain the very first header (with the verb, address and HTTP version), such as: GET /favicon.ico HTTP/1.1

Some other features that I would love to see (ordered by estimated complexity) are:

• Timing information (such as time to first byte / total duration and so on)... but I suspect that this might already exist in the Performance tab
• Case-insensitive filtering (this already seems to work for header filtering, but not body filtering using body*= for instance)
• Request replay (ie. manually re-submitting a captured request, using the same request headers / body)
• Dark-theme for night-owls :)
• Element / key count for JSON request / response bodies (ie. a quick visual cue as to how many items there are in an array / collection)
• In-flight request / response information which shows a sample of the information being transferred - and / or a total byte count transferred so far
• Configureable body content-dropping (this might help performance when downloading large files, by only parsing the first x bytes from the response)
• Optionally visible CONNECT requests which shows the SSL/TLS version used, and other certificate / cipher information
• Request editor / composer (Create a new equest - or edit a captured one - manually and re-send it
• Request / response breakpoint(s) (allowing the interception of requests / response before being sent to the intended destination to allow manual editing)

Re. point 6 above, I tried downloading a 1GB file from here using Firefox just to see how the tool handles it, and while it did log the request, the response was never marked as completed (and hence no response headers / body) where shown at all.

Any of the above can be opened up as separate issues to ease tracking, but I thought I'd give a quick run-through of the feature requests that I would love to see.

---

Does this affect you too? Click the +:smiley: icon in the top right and add a :+1: on issues to help decide where HTTP Toolkit goes next, or go take a look & vote on the most popular ideas so far.

[pimterry]

Hi @TrapperHell, great to hear you're finding HTTP Toolkit useful! To work through your points here:

What I'm really missing at the moment is a raw text display of the complete request (headers + body) and complete response content (headers + body) that is easily copyable.

Agreed. Right now, Pro users can get a raw HTTP/1.1 export of the request data from the Export section when a request is selected, but that doesn't include the response data.

In future, I think I'd definitely like to be able to export both together easily (tracked in #127). It's likely that that'll be supported through a right-click context menu to access all export formats (#162), in addition to allowing selection of multiple requests at once to export those in a batch (#76). That's not available yet, and it'd be a Pro feature, but it's not an especially difficult addition I think.

Timing information (such as time to first byte / total duration and so on)

The total round trip time is indeed available in the performance section for Pro users, but that's just the overall time (the rest of the performance section is explaining caching rules based on the headers that were seen, and testing compression performance for the body used).

More detailed timing information is actually already recorded and available in the UI data model, it's just that it's not exposed right now, but only because the design to do so isn't defined yet.

Out of interest, what is it that you're looking for specifically in that data? Can you explain your use case in more detail?

Case-insensitive filtering (this already seems to work for header filtering, but not body filtering using body*= for instance)

Ah, that's interesting. This is potentially a bit expensive though, depending on how its done. Bodies can be large, so converting all of them to lowercase is potentially time consuming. We could cache the lowercased data for matching, but that pushes memory usage up.

I think potentially we could use the built-in case-insensitive regex matching though, which case-insensitively matches as it goes, without converting all the data up front, and which should be very quick since it's implemented natively... I've made a new issue to track that: #228

Request replay (ie. manually re-submitting a captured request, using the same request headers / body)
Request editor / composer (Create a new request - or edit a captured one - manually and re-send it)

Agreed! Definitely want to do this eventually, it's tracked in #48

Dark-theme for night-owls :)

This is available - it's a Pro feature.

Element / key count for JSON request / response bodies (ie. a quick visual cue as to how many items there are in an array / collection)

You mean like an inline count at the start of each array/object within the body editor? That's a good idea, I haven't heard that before but I can totally see how it'd be useful.

The editor is built on Monaco which supports a wide variety of extensions we can use to inject custom logic in here. This could be implemented as an inlay hint to show metadata inline (shown like TypeScript param metadata in VS code: https://code.visualstudio.com/docs/languages/typescript#_inlay-hints) or as a CodeLens to show floating separate lines before the array/object itself (like the 'First line' marker here: https://microsoft.github.io/monaco-editor/playground.html#extending-language-services-codelens-provider-example). I've opened an issue: #229

In-flight request / response information which shows a sample of the information being transferred - and / or a total byte count transferred so far

This would be good too, agreed. That data isn't currently available in the UI yet (there's no easily available data on progress of ongoing responses until they complete) but it would be good. I'm working on websockets very soon (#36) and the UI to display ongoing streams is likely to relate to this, so I'll keep an eye out for ways to handle both at once.

Configureable body content-dropping (this might help performance when downloading large files, by only parsing the first x bytes from the response)

True! This is somewhat configuration internally in Mockttp already (https://github.com/httptoolkit/mockttp/blob/a54b9960d8ddf0df071f7d475de3777e3418e771/src/server/mockttp-server.ts#L89) which has some behaviour to handle this, but as you say it doesn't work well for many cases right now. That's tracked in #143.

Optionally visible CONNECT requests which shows the SSL/TLS version used, and other certificate / cipher information

This is interesting, I haven't heard that suggestion before. Can you explain why you want this information, and how you'd use it?

I'm open to adding this, but it's a careful balance to providing maximum information without overcomplicating the UI & UX.

Request / response breakpoint(s) (allowing the interception of requests / response before being sent to the intended destination to allow manual editing)

This actually already exists, and it's not a Pro feature! You can breakpoint and manually edit requests by creating a rule on the mock tab that matches your target traffic and selecting on the "Pause the {request/response/both} to manually edit it" options.

---

I'll keep an eye out for ways to implement these soon, but as you can imagine I'm already pretty busy and the upcoming Docker release and then websocket support is currently top of the list. If you're interested in any of the issues linked above you can 👍 on those to vote for them, and/or subscribe to notifications of the issue via GitHub to hear when there's progress.

In the meantime though, if you're really interested in this, all of HTTP Toolkit is open source and HTTP Toolkit Pro is totally free for all contributors. You're very welcome to take a shot at whichever of these would be most valuable to you. There's contribution guidelines here: https://github.com/httptoolkit/httptoolkit#contributing-directly and I think most things you're talking about would be changes in the UI here: https://github.com/httptoolkit/httptoolkit-ui. Let me know if you want any pointers in the right direction.

[TrapperHell]

Thanks for all the detail you've went through and for so meticulously tracking the issues. I've also taken the liberty to upvote the mentioned issues which I have deemed to come in really handy (personally).

Before providing context to the two questions you've asked, I will preface by saying that I do a lot of work both creating and interacting with APIs (both professionally and for my hobby projects).

Timing information (such as time to first byte / total duration and so on)

This is very use-case dependent, but I do find myself checking the API timings from time to time to measure performance / latency etc - especially when mixing in other factors (such as compression / caching). This is besides stress-testing, just to get a general feel of what I'm working with.

Optionally visible CONNECT requests which shows the SSL/TLS version used, and other certificate / cipher information

Once again, this is very use-case specific, and in this case it's even rarer that I find myself looking at these stats. The main use-case for this is when I'm interacting with APIs that work with a very specific set of certificates, and I would need to debug what's going on. And yes I do feel your concern for littering the UI for something which is very remotely used. I suppose this could come in handy when the right-click context menu is implemented, in which case it could be hidden neatly inside.

As for the rest of the comments:

The editor is built on Monaco which supports a wide variety of extensions we can use to inject custom logic in here. This could be implemented as an inlay hint to show metadata inline [...]

Yup, that would work really well imo. Gets the message across without being obtrusive.

This actually already exists, and it's not a Pro feature!

My bad on that one. I took a quick glance at the Mock tab, and noticed the Pro item in the dropdown, and discounted that to being entirely locked. Now that I look back at it, I can see how much of an idiot I was.

You're very welcome to take a shot at whichever of these would be most valuable to you.

On a side-note I really wanted to thank you for open-sourcing this project, and I love the philosophy with which you've created this project. To be quite honest TypeScript (+ nodejs arch) isn't my goto language, I'm actually surprised that you managed to achieve all this through node actually). At the time I'm pretty swamped with my own personal projects, so I totally understand that you might not have time for these either at the moment. That being said, I'll keep this in mind, and maybe I'll find time to venture into this project and ease myself into a simpler task, such as the regex-insensitive task.

Thanks again

[pimterry]

I do find myself checking the API timings from time to time to measure performance / latency etc - especially when mixing in other factors (such as compression / caching). This is besides stress-testing, just to get a general feel of what I'm working with.

Makes sense! In that case I imagine it's mostly the top-line numbers that are interesting, rather than the low-level details. I'll see if there's an easy way to expose a little more of those (#232).

Optionally visible CONNECT requests which shows the SSL/TLS version used, and other certificate / cipher information

Once again, this is very use-case specific, and in this case it's even rarer that I find myself looking at these stats. The main use-case for this is when I'm interacting with APIs that work with a very specific set of certificates, and I would need to debug what's going on.

Ah, I see, that's helpful thanks. I'll keep an eye on this, but I think it's probably out of scope. HTTP Toolkit has to draw the line somewhere. I think this low-level info isn't that relevant for most HTTP-focused use cases, and in the cases where it is relevant you're probably debugging at a lower level, where tools like Wireshark are more appropriate. Wireshark can happily show you the certificates used in TLS handshakes with no problem, and you can use that at the same time as HTTP Toolkit if you want to do HTTP & TLS debugging together.

I'll keep thinking about it, and I'm open to discussion, but for now I think it's probably not worth the complexity.

That being said, I'll keep this in mind, and maybe I'll find time to venture into this project and ease myself into a simpler task, such as the regex-insensitive task.

Fair enough, I totally understand. If you do decide you want to look into that, the relevant code for that filtering starts here: https://github.com/httptoolkit/httptoolkit-ui/blob/a85436746c6bdaaf308d549b97a0d43d11e932a3/src/model/filters/search-filters.ts#L1082-L1109

On a side-note I really wanted to thank you for open-sourcing this project, and I love the philosophy with which you've created this project.

Thanks, great to hear you're finding it useful! 😄 I think everything left here is now covered in standalone issues, so I'll close this issue, but do feel free to get in touch if you have any more feedback or hit any issues in future. Thanks for all the suggestions!