Creating policies does not scan existing projects for violations

[rkg-mm]

Current Behavior

When creating a new Policy, I would expect the system to apply the policy to all active projects. However, it seems this is not the case. Existing projects do not get any policy violations.

Steps to Reproduce

1. Create some projects
2. Create a new policy that will trigger violations on some of the projects
3. Wait a while
4. look for policy violations - you won't find any

Expected Behavior

I would expect the system to schedule a policy analysis on all existing projects once a new policy is created (or updated, including changes to the scope of a policy), and therefore shortly after policy creation some violations should appear, but this is currently not happening. Combined with the lack of a way to trigger this manually (#3951) one lacks a way to see the effect of a policy directly.

Dependency-Track Version

4.13.0-SNAPSHOT

Dependency-Track Distribution

Executable WAR

Database Server

H2

Database Server Version

No response

Browser

Mozilla Firefox

Checklist

• I have read and understand the contributing guidelines
• I have checked the existing issues for whether this defect was already reported

[nscuro]

In my view, this is mostly a performance concern.

Re-evaluating all projects becomes increasingly more expensive as the portfolio grows.

The UI is also a bit over-sensitive, in that it auto-saves changes after a short debounce period (I think 750ms). This will need to change, otherwise we'd DoS the system with too many re-evaluations. Saving needs to become an explicit action.

I also don't think that every change to a policy necessitates a full re-evaluation. In most cases, knowing that the change will eventually take effect is enough. So IMO if we offer a full re-evaluation, it should be an explicit action users have to trigger, rather than implicit via policy updates.

Separately, I do agree that testing is inconvenient at the moment. I'd envision a test dialog of some sort, which allows users to pick one or more projects to "dry run" the policy for. This would limit the system impact, while still allowing users to observe the effect of their policy.

[valentijnscholten]

The "Evaluate" button could trigger an evaluation only of that policy to save on resources needed? I think it's not an uncommon usecase to create a policy because you want to know "instantly" (within reasonable amount of time) if there are any violations.

[rkg-mm]

Just one use case that came in to my mind: If I want to blacklist a compromised component, I could do this via policy to inform all teams automatically. But not if the policy does not trigger immediately.

[LesSyner]

Running full re-evaluation after each change would be overkill (especially when any change is saved instantly after 1s or so). But it's natural that after making changes to policies you expect to see impact. Use cases above mention mostly small impact but the same bug applies if I create new policy with high impact but don't see any impact. This bug also makes whole workflow cumbersome in new setup - you have to prepare final set of policies and only then start creating projects and uploading SBOM files since all policies changes afterwards will have ni impact. A bit weird (to say the least).
My opinion is that at least there should be way (like button on the policies management page) to re-evaluate all projects to be used once you finished editing policies. I understand that inactive projects can be omitted but it should re-evaluate at least all active projects.