Rust: regenerate MaD files using DCA

[redsun82]

• rust autogenerated models now use the DCA strategy
• models were regenerated from a recent DCA run
• the bulk model generator got some changes:
  • the configuration files are now in YAML format, which is terser and more consistent with how we generally configure stuff
  • running the DCA strategy the generator will now take the last DB artifact for each project, which makes it compatible to run against comparing DCA runs
  • downloads from DCA are now run in parallel (up to a maximum of 8 workers), which scales much better with respect to the number of sources
  • the bulk generator cleans up extracted DB locations, which makes it rerunnable without any manual cleanup
  • the generator can now by run directly on POSIX, without needing an explicit python invocation

[Copilot]

Pull Request Overview

This PR updates the bulk model generator to use the DCA strategy for regenerating MaD files, switches configuration from JSON to YAML, and enhances parallelism and cleanup in the Python script.

• Migrate bulk generation config files from JSON to a terser YAML format.
• Refactor bulk_generate_mad.py to add a generic run_in_parallel helper for cloning and downloading in parallel, with cleanup of old artifacts.
• Regenerate all Rust QL test expected files based on the new DCA outputs.

Reviewed Changes

Copilot reviewed 70 out of 70 changed files in this pull request and generated 1 comment.

File	Description
misc/scripts/models-as-data/bulk_generate_mad.py	Add run_in_parallel, parallel DCA downloads, YAML parsing, cleanup logic
rust/misc/bulk_generation_targets.yml	New YAML config replacing JSON targets for Rust bulk generation
cpp/bulk_generation_targets.yml	New YAML config replacing JSON targets for C++ bulk generation
Various .expected files under rust/ql/test	Regeneration of QL test expectations to reflect new DCA outputs

Comments suppressed due to low confidence (1)

misc/scripts/models-as-data/bulk_generate_mad.py:115

• The generic type parameters T and U are used in the function signature but not defined; add TypeVar definitions such as T = TypeVar('T') and U = TypeVar('U') before their use.

def run_in_parallel[T, U](

[Copilot]

[nitpick] Exiting from within a utility function (via sys.exit in on_error handlers) can make the logic harder to test or reuse; consider returning errors and handling exit at the top level instead.

Suggested change

	dirs = run_in_parallel(
	failed = run_in_parallel(

[geoffw0]

Looks great! Some comments / discussion, one test annotation needs fixing.

[geoffw0]

This is a nice simple list, once everything is merged and stable I'll add a bunch more targets to it.

[redsun82]

one thing to keep in mind is that at the moment this list needs to be topologically ordered with respect to dependencies (so later additions should depend on earlier ones and not the other way around). Possibly worth a comment here, now that this is yaml

[redsun82]

also, just so you know, you can tweak what gets generated with any of

with-sinks: false
with-sources: false
with-summaries: false

(all are true by default)

[geoffw0]

What are the expected use cases for those three options?

[redsun82]

I don't really know, but you can ask Mathias once he's back from his PTO, two of them are used for the C++ generated models

[geoffw0]

My guess is there are certain libraries that produce a lot of inaccurate models of one type but not the others, and this gives us some additional control. @MathiasVP ? (no rush, not blocking this PR)

[MathiasVP]

The reason is simply that C++ doesn't yet autogenerate sources and sinks (for a couple of reasons, but mainly because I didn't bother to set that properly up yet). The MaD generator script (which this script invokes under the hood) already provides these hooks to configure which kinds of models are generated, so I just lifted those hooks to this script in #19627

[paldepind]

Looks really great!

A few comments and I think you need to run black again as there's a few formatting changes from that.

[redsun82]

@paldepind Can I get a reapproval after resolving a merge conflict? 🙌