max_tokens missing in Single object

[dapascual]

Hello, when running garak with a custom function I get this error:
'Single' object has no attribute 'max_tokens'
I fixed it locally by setting in the class Single the parameter max_tokens to e.g., 150 here: https://github.com/NVIDIA/garak/blob/main/garak/generators/function.py#L62. It'd be great if you could fix that in the repository.
Thanks and congratulations for your work!

[leondz]

generators.functions.Single should load max_tokens from DEFAULT_PARAMS set in generators.base.Generator when self._load_config is called (https://github.com/NVIDIA/garak/blob/main/garak/generators/function.py#L72), could you share your code?

[dapascual]

Yes, in a file called my_functions.py in my_module/ I have a simple function, e.g.,

from typing import List, Union

def identity_function(prompt:str, **kwargs)->List[Union[str, None]]:
    return ["Identity"]

Then in another file called garak_cli.py I do:

import garak
import garak.cli

garak.cli.main("--model_type function  --model_name my_module#identity_function --generations 1".split())

Then, when I run python garak_cli.py it runs the first few probes, and after the dan.STAN probe I get this error:

Traceback (most recent call last):
  File "my_path/garak_cli.py", line 21, in <module>
    garak.cli.main("--model_type function "
  File "my_path/.venv/lib/python3.11/site-packages/garak/cli.py", line 594, in main
    command.probewise_run(
  File "my_path/.venv/lib/python3.11/site-packages/garak/command.py", line 237, in probewise_run
    probewise_h.run(generator, probe_names, evaluator, buffs)
  File "my_path/.venv/lib/python3.11/site-packages/garak/harnesses/probewise.py", line 107, in run
    h.run(model, [probe], detectors, evaluator, announce_probe=False)
  File "my_path/.venv/lib/python3.11/site-packages/garak/harnesses/base.py", line 123, in run
    attempt_results = probe.probe(model)
                      ^^^^^^^^^^^^^^^^^^
  File "my_path/.venv/lib/python3.11/site-packages/garak/probes/base.py", line 219, in probe
    attempts_completed = self._execute_all(attempts_todo)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_path/.venv/lib/python3.11/site-packages/garak/probes/base.py", line 197, in _execute_all
    result = self._execute_attempt(this_attempt)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "my_path/.venv/lib/python3.11/site-packages/garak/probes/base.py", line 155, in _execute_attempt
    self._generator_precall_hook(self.generator, this_attempt)
  File "my_path/.venv/lib/python3.11/site-packages/garak/probes/divergence.py", line 72, in _generator_precall_hook
    if self.override_maxlen and self.generator.max_tokens < self.new_max_tokens:
                                ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Single' object has no attribute 'max_tokens'

[leondz]

Thank you, will take a look

[ls-omar-nbou]

Hello there, I got exactly the same error when running a custom function:

import requests

def chatbot_gemini_interface(prompt:str, **kwargs)->str:
     form_data = {"prompt": prompt}
     response = requests.post("<URI_LOCALHOST>",data=form_data)
     return [response.text]

import garak
import garak.cli
import chatbot_gemini_interface

garak.cli.main("--model_type function --model_name chatbot_gemini_interface#chatbot_gemini_interface -d shields --verbose".split())

seems to be happening when loading and running the probe garak.probes.divergence.Repeat.

Keep up the great work !