Secrets-plugin: redact fails when redact-mode=all and more than 1 secret in prompt causing leak of secret

[mariovitale1979]

Describe the bug

When using the Secrets plugin in 'all' redact-mode AND using a prompt with more than 1 secret, the sanitized prompt will still contain clear text secrets.

To Reproduce

Using the LLM Guard playgroun (https://huggingface.co/spaces/protectai/llm-guard-playground) , or llm-guard library or the Swagger API, call the "scan_prompt" function or the /analyze/prompt API call with a prompt like this:
"This is a prompt example containing fake 2 secrets, ex sk-ZYX987654321abcABC12T3BlbkFJqwertyyiop1234567890 and ghp_d2PHL8sMWUfBPZVNvHtAr4co4Zfy2Y3RB42O"

The sanitized prompt will look like this:
This is a prompt example containing fake 2 secrets, ex ****** and ghp_d2PHL8sMWUfBPZVNvHtAr4co4Zfy2Y3RB42O************ and ghp_d2PHL8sMWUfBPZVNvHtAr4co4Zfy2Y3RB42O

The 1st secret is correctly redacted but the 2nd is not. Even worse, it's still in cleartext and even duplicated somehow

Expected behavior

The expected behaviour is that all secrets should be redacted no matter if there is 1 or multiple in a prompt or output.

Screenshots

[mariovitale1979]

I think I found the rootcause for this issue.
The problem lies in the 'scan() function in secrets.py, more specifically in this section: https://github.com/protectai/llm-guard/blob/main/llm_guard/input_scanners/secrets.py#L475-L492

This section will build a sanitized prompt string based on the different secret types that where found. In a previous step a scan of the prompts was done returning a 'secrets' list that contains information about what secret type was found and in what position and length it can be found in the string.
That information is then used to build the sanitized prompt by replacing these parts with a redacted value and here lies the problem. When using the redact-mode=all, the secret is replaced by 6-digits '*', thus changing the eventual string length of the newly build sanitized prompt. This works fine with the 1secret but when processing the 2nd secret and looking up it's start-index, this value is not valid anymore.

I was able to write patch that works for me.
The patch will do the secret replacement operation starting from the back which should fix the problem desribed above

secrets-redact.patch