Clarify network policy: does "web search off" mean full network isolation, or only the web-search tool?

[iwanttobeagoodman]

Title: Clarify network policy: does "web search off" mean full network isolation, or only the web-search tool?

Body:

Summary

The paper states:

Web search is turned off for the benchmark to keep evaluations reproducible.

It's not clear whether this means (a) only the web-search MCP tool is disabled, or (b) the whole environment/container is network-isolated (no outbound egress at all). These have very different reproducibility implications, because other tools — notably the code execution server — can make arbitrary outbound requests independently of the web-search tool.

Could the maintainers clarify the intended network policy for benchmark runs, and (ideally) document/enforce it?

Why this is ambiguous in the repo

1. code_exec can reach the network regardless of web search. The code execution sandbox (mcp_servers/code/.../sandbox_fs.c, injected via LD_PRELOAD) only intercepts filesystem syscalls — it blocks paths /app:/.apps_data:/proc:/sys but does not touch socket/connect/getaddrinfo. The env allowlist in utils/sandbox.py deliberately preserves HTTP_PROXY/HTTPS_PROXY/NO_PROXY and TLS CA vars, with a comment: # --- Egress proxy (dropping these kills all network access) ---, and the file notes the sandbox "permits pip install, network downloads." So an agent can curl/urllib any URL from code_exec even with the web-search tool turned off.

   I reproduced this exactly (same debian:trixie-slim base, same gcc ... sandbox_fs.c compile, same LD_PRELOAD + SANDBOX_BLOCKED_PATHS invocation): reading /app was correctly Permission denied (sandbox active), but https://example.com → 200, https://api.github.com/zen → 200, python3 urllib.urlopen(...) → 200. i.e. the tool/sandbox layer does not block egress.

2. INTERNET_ENABLED is documented but not enforced in this repo. 8 of 9 server READMEs document an arco.toml runtime flag INTERNET_ENABLED = "false" with UI label "Internet access" / "Allow the MCP server to make outbound network requests" ("Network policy flag"). However:

   • INTERNET_ENABLED appears only in README.md files — zero references in any .py/.c/.sh/.toml source, and there is no actual arco.toml in the repo. So enforcement (if any) lives in the external ArCo / RL Studio deployment layer, not here.
   • mcp_servers/documents/README.md omits INTERNET_ENABLED entirely (no value, no schema, no bullet), which suggests the docs are inconsistent/templated.

Questions

1. For the published benchmark numbers, is the environment fully network-isolated (no egress), or is only the web-search tool disabled while code_exec etc. retain outbound access?
2. Is INTERNET_ENABLED=false actually enforced at runtime for benchmark containers, and by what (network namespace / firewall / egress proxy allowlist)?
3. If egress is allowed, how is reproducibility guaranteed given code_exec can fetch arbitrary URLs / pip install at runtime?
4. Could the intended policy be documented in one place (and the documents README be brought in line with the other 8)?

Thanks!