SWEBench Evaluation

[shawnlewis]

Fast SWEBench evaluations for programmer, logged with Weave.

Initial evaluation results on SWEBench verified here: https://wandb.ai/shawn/weavedev-programmereval1/weave/traces

Design:

• Build all SWEBench instance images on a server
• Run cmserver.py on the server to server container access
• Setup programmer with RemoteContainerToolContext so it works in containers on the server
• Use programmer/swebench/evaluate.py to run eval with Weave, and capture results.

[socket-security]

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
pypi/[email protected]	environment, filesystem, network, shell	0	511 kB	aiordache, ccrone, dev-tooling-team, ...4 more
pypi/[email protected]	environment, filesystem, network, shell	0	3.83 MB	tiangolo
pypi/[email protected]	environment, eval, filesystem, network, shell, unsafe	0	879 kB	mdurant, tomaugspurger
pypi/[email protected]	Transitive: environment, eval, filesystem, network, shell, unsafe	+4	811 kB
pypi/[email protected]	environment, filesystem, network, shell	0	54.4 kB	craigie
pypi/[email protected]	environment, eval, filesystem, network, shell Transitive: unsafe	+268	373 MB	carlosej, johny-nlp

View full report↗︎