pkg/terminal: allow custom starlark commands to be used in on prefix

[derekparker]

Fixes #4160

[aarzilli]

PS. if we constrain the things that an on command can do to things that don't resume execution (and don't change variables) all these problems go away and we can even leave it in printcontextThread, however the motivation for the original issue was specifically about resuming execution.
Arguably that maybe should be done with starlark in cond rather than in on but that would involve running starlark server side, which is a bigger refactor.

[aarzilli]

Arguably that maybe should be done with starlark in cond rather than in on but that would involve running starlark server side, which is a bigger refactor.

PPS. Thinking more about this having a reduced server-side version of the starlark interpreter that can only call a small subset of our API (eval, stacktraces, goroutine lists...) could be useful.

[derekparker]

Arguably that maybe should be done with starlark in cond rather than in on but that would involve running starlark server side, which is a bigger refactor.

Yeah, I had this thought originally too. This was simpler, but also I think it's a useful feature outside the original request.

[derekparker]

Marking as draft as there is an edge case I thought of that I'd like to fix before another review.

[derekparker]

Alright, updated to handle more situations explicitly.

The flow now is:

• Program runs, hits a breakpoint w/ 3 custom commands.
• The 2nd of the 3 commands causes the program to continue execution.
• The 3rd custom command is canceled (does not run)
• Any breakpoint hit during continued execution will have its custom commands (if any) execute, following the same rule as above.

The code handles multiple layers of this invalidation logic by maintaining a stack of booleans on the Term struct any time we execute custom commands.

One caveat that still remains is raw_command from within a starlark command will not follow these rules. This seems ok to me because raw_command comes with its own set of caveats, and if someone is using that chances are they know what they're doing and why they are using that over dlv_command. Either way, I'm happy to be more explicit there in the docs or have any raw_command invalidate the custom commands. We could also throw an error or print a warning if raw_command is used in a command with the on prefix.

[aarzilli]

LGTM