T5 Tokenzier not load with AttributeError: add_special_tokens conflicts with the method add_special_tokens in T5Tokenizer

[QuantumStaticFR]

System Info

• transformers version: 4.48.2
• Platform: Linux-5.10.0-33-cloud-amd64-x86_64-with-glibc2.31
• Python version: 3.12.0
• Huggingface_hub version: 0.28.1
• Safetensors version: 0.5.2
• Accelerate version: not installed
• Accelerate config: not found
• PyTorch version (GPU?): 2.6.0+cu118 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: No
• GPU type: NVIDIA A100-SXM4-40GB

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Running this script should reproduce the error

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_name = "StonyBrookNLP/teabreac-preasm-large-drop"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

It fails with

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[6], line 2
      1 model_name = "StonyBrookNLP/teabreac-preasm-large-drop"
----> 2 tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
      3 model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
      4 # enable_digit_tokenization(tokenizer)

File ~/my/path/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py:934, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    930     if tokenizer_class is None:
    931         raise ValueError(
    932             f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
    933         )
--> 934     return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    936 # Otherwise we have to be creative.
    937 # if model is an encoder decoder, the encoder tokenizer class is used by default
    938 if isinstance(config, EncoderDecoderConfig):

File ~/my/path/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:2036, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, cache_dir, force_download, local_files_only, token, revision, trust_remote_code, *init_inputs, **kwargs)
   2033     else:
   2034         logger.info(f"loading file {file_path} from cache at {resolved_vocab_files[file_id]}")
-> 2036 return cls._from_pretrained(
   2037     resolved_vocab_files,
   2038     pretrained_model_name_or_path,
   2039     init_configuration,
   2040     *init_inputs,
   2041     token=token,
   2042     cache_dir=cache_dir,
   2043     local_files_only=local_files_only,
   2044     _commit_hash=commit_hash,
   2045     _is_local=is_local,
   2046     trust_remote_code=trust_remote_code,
   2047     **kwargs,
   2048 )

File ~/my/path/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:2276, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, token, cache_dir, local_files_only, _commit_hash, _is_local, trust_remote_code, *init_inputs, **kwargs)
   2274 # Instantiate the tokenizer.
   2275 try:
-> 2276     tokenizer = cls(*init_inputs, **init_kwargs)
   2277 except import_protobuf_decode_error():
   2278     logger.info(
   2279         "Unable to load tokenizer model from SPM, loading from TikToken will be attempted instead."
   2280         "(Google protobuf error: Tried to load SPM model with non-SPM vocab file).",
   2281     )

File ~/my/path/lib/python3.12/site-packages/transformers/models/t5/tokenization_t5.py:189, in T5Tokenizer.__init__(self, vocab_file, eos_token, unk_token, pad_token, extra_ids, additional_special_tokens, sp_model_kwargs, legacy, add_prefix_space, **kwargs)
    186 self._extra_ids = extra_ids
    187 self.add_prefix_space = add_prefix_space
--> 189 super().__init__(
    190     eos_token=eos_token,
    191     unk_token=unk_token,
    192     pad_token=pad_token,
    193     extra_ids=extra_ids,
    194     additional_special_tokens=additional_special_tokens,
    195     sp_model_kwargs=self.sp_model_kwargs,
    196     legacy=legacy,
    197     add_prefix_space=add_prefix_space,
    198     **kwargs,
    199 )

File ~/my/path/lib/python3.12/site-packages/transformers/tokenization_utils.py:435, in PreTrainedTokenizer.__init__(self, **kwargs)
    432 self._added_tokens_encoder: Dict[str, int] = {k.content: v for v, k in self._added_tokens_decoder.items()}
    434 # 4 init the parent class
--> 435 super().__init__(**kwargs)
    437 # 4. If some of the special tokens are not part of the vocab, we add them, at the end.
    438 # the order of addition is the same as self.SPECIAL_TOKENS_ATTRIBUTES following `tokenizers`
    439 self._add_tokens(
    440     [token for token in self.all_special_tokens_extended if token not in self._added_tokens_encoder],
    441     special_tokens=True,
    442 )

File ~/my/path/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1407, in PreTrainedTokenizerBase.__init__(self, **kwargs)
   1405 for key in kwargs:
   1406     if hasattr(self, key) and callable(getattr(self, key)):
-> 1407         raise AttributeError(f"{key} conflicts with the method {key} in {self.__class__.__name__}")
   1409 self.init_kwargs = copy.deepcopy(kwargs)
   1410 self.name_or_path = kwargs.pop("name_or_path", "")

AttributeError: add_special_tokens conflicts with the method add_special_tokens in T5Tokenizer

Expected behavior

I expected the tokenizer to load. Two similar issues were raised before but they did not solve my problem.

1. Regression in tokenizer loading #33453
2. [BUG] CodeGen 2.5 Tokenizer cannot be initialized anymore salesforce/CodeGen#94

[Rocketknight1]

Hi @QuantumStaticFR, I believe this is because of a key in the tokenizer_config.json for that model that is no longer supported. Please raise the issue with the repo owners!

[QuantumStaticFR]

I'll tell them their config is no longer supported by huggingface and request them to update it?

[Rocketknight1]

Yeah, I think that's the right strategy. It's also possible that this is caused by the tokenizer needing to pop that kwarg correctly - cc @ArthurZucker ?

[ArthurZucker]

Yeah, but I think we should support serializing and de-serializing this argument. It's a little bit annoying because we have to check the arg all the time, but let me open a PR (not the first time this happens)