Madlad400 support #1416

irshadbhat · 2024-01-09T05:48:36Z

Model description

T5 based model for Machine translation.

Open source status

The model implementation is available
The model weights are available

Provide useful links for the implementation

https://huggingface.co/jbochi/madlad400-3b-mt#training-details

System Info

text-generation-inference 1.3.0
python 3.10

Information

Docker

Tasks

An officially supported command

Reproduction

The model loaded proporly but I am getting incorrect results. I want to know if the tgi 1.3.0 supports Madlad400? Thanks a lot.

Expected behavior

Madlad400 support

Narsil · 2024-01-10T16:59:09Z

Is flan-t5 working correctly for you ? https://huggingface.co/google/flan-t5-base.

We have integration-tests for t5, therefore this model should work, but we have no idea how this particular checkpoint was created, it's very possible that the model was wrongly converted: https://github.com/huggingface/text-generation-inference/blob/main/integration-tests/models/test_t5_sharded.py

irshadbhat · 2024-01-11T18:14:23Z

Have tested flan-t5-xl and its working fine. The results are exactly as expected.

github-actions · 2024-02-11T01:45:44Z

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

jhargraveiii · 2025-01-16T16:59:45Z

We are seeing exactly the same thing with madlad400. It seems to work but produces very poor results compared to running the same model with standard transformers code. Perhaps its the pipeline type? We use "translation". flan-t5 doesn't AFAICT.

jhargraveiii · 2025-01-16T18:37:03Z

Can we repopen this issue? We've tested on the latest 3.0.1

josh-at-straker · 2025-01-19T19:23:00Z

Hey team - Jumping on this as a 'me-too'.

We have issue on MadLad (all the google released versions on HF) and our fine-tuned variants. I'm looking for other similar t5 translation models to test with, but a new translations pipeline in TGI would be a game changer.

The model loaded correctly, however I am getting incorrect results back from the model.

jrc2139 · 2025-03-06T23:28:37Z

i had an interesting find while running madlad through tgi 3.1.1

curl 127.0.0.1:8081/generate \                                                                                                                       
    -X POST \
    -d '{
  "inputs":"<2fr> Les Holden (6 March 1895 – 18 September 1932) was a fighter ace of World War I. He joined the Australian Light Horse in May 1915, serving in Egypt and France. In December 1916, he volunteered for the Australian Flying Corps and qualified as a pilot. As a member of No. 2 Squadron, he gained the sobriquets \"Lucky Les\" and \"the homing pigeon\" after a series of incidents where he limped back to base in bullet-riddled aircraft. Holden was awarded the Military Cross, achieved five aerial victories, finishing the war as an instructor with No. 6 (Training) Squadron in England, earning the Air Force Cross. After leaving the Australian Flying Corps in 1919, he became a manager at Holdens Motor Body Builders. He joined the part-time Citizen Air Force before starting an air service as a commercial pilot. In 1929, he located Charles Kingsford Smith and Charles Ulm in the north-west Australian desert after the pair were reported missing. Holden was killed in a passenger plane crash in Australia."
}' \
    -H 'Content-Type: application/json'

produces this

{
    "generated_text": "Des Heus demeurait dans cette station jusqu’en  управления d’utilité publique trois হায়খিlstelya іншых pour le roi kerajaan Siol son prénom, Holden était un as de la Première Guer falintu mondiale. Il rejoignit l’Australian Light Horse en mai 19 Kamuktaindetzeraire Jeruzalemu 1 ⵓⵉDis demeurait dans cette station jusqutogailles quatre mois de la Première Guer rikicè Afrika. En décembre 1916, il livra sa tâche volontaire au Corps de l’Aviation Australien et obtint son diplôme de Pilot: membre de l’Escadron Noегін kovića, devint l’Apparemment un astronaute et devenu membre de la Royal Navy. En novembre 1917, il quitta la Sioux National Ghost Conservancy ( kibutz Maclean, Bullenberg District,κληρονόμος Ενεργειών Εθνικής payauHubezathair Peredmen Enfas. GháfomiÈ, mishtos Kil. gusht.улалkiso. b.gnanta. n'a jamais dévoilé"
}

but adding "return_full_text": true

curl 127.0.0.1:8081/generate \
  -X POST \
  -d '{
  "inputs":"<2fr> Les Holden (6 March 1895 – 18 September 1932) was a fighter ace of World War I. He joined the Australian Light Horse in May 1915, serving in Egypt and France. In December 1916, he volunteered for the Australian Flying Corps and qualified as a pilot. As a member of No. 2 Squadron, he gained the sobriquets \"Lucky Les\" and \"the homing pigeon\" after a series of incidents where he limped back to base in bullet-riddled aircraft. Holden was awarded the Military Cross, achieved five aerial victories, finishing the war as an instructor with No. 6 (Training) Squadron in England, earning the Air Force Cross. After leaving the Australian Flying Corps in 1919, he became a manager at Holdens Motor Body Builders. He joined the part-time Citizen Air Force before starting an air service as a commercial pilot. In 1929, he located Charles Kingsford Smith and Charles Ulm in the north-west Australian desert after the pair were reported missing. Holden was killed in a passenger plane crash in Australia.",
  "parameters": {
    "return_full_text": true
  }
}' \
  -H 'Content-Type: application/json'

produces a correct translation (however the original text is apart of the response and not really able to parse it out)

{
  "generated_text": "<2fr> Les Holden (6 March 1895 – 18 September 1932) was a fighter ace of World War I. He joined the Australian Light Horse in May 1915, serving in Egypt and France. In December 1916, he volunteered for the Australian Flying Corps and qualified as a pilot. As a member of No. 2 Squadron, he gained the sobriquets \"Lucky Les\" and \"the homing pigeon\" after a series of incidents where he limped back to base in bullet-riddled aircraft. Holden was awarded the Military Cross, achieved five aerial victories, finishing the war as an instructor with No. 6 (Training) Squadron in England, earning the Air Force Cross. After leaving the Australian Flying Corps in 1919, he became a manager at Holdens Motor Body Builders. He joined the part-time Citizen Air Force before starting an air service as a commercial pilot. In 1929, he located Charles Kingsford Smith and Charles Ulm in the north-west Australian desert after the pair were reported missing. Holden was killed in a passenger plane crash in Australia.Il rejoint l'Australian Light Horse en mai 1915, servant en Égypte et en France. En décembre 1916, il se porte volontaire pour le Australian Flying Corps et se qualifie comme pilote. En tant que membre du 2e Escadron, il obtient les surnoms de « Lucky Les » et « le pigeon de retour » après une série d'incidents où il boite à la base dans des avions couverts de balles. Holden reçoit la Military Cross, réalise cinq victoires aériennes, termine la guerre comme instructeur avec le 6e Escadron (d'entraînement) en Angleterre, gagnant la Air Force Cross. Après avoir quitté l'Australian Flying Corps en 1919, il devient manager chez Holdens Motor Body Builders. Il rejoint la Citizen Air Force à temps partiel avant de commencer un service aérien comme pilote commercial. En 1929, il retrouve Charles Kingsford Smith et Charles Ulm dans le dé"
}

the only references i can find in the code are:

UPDATE:
it seems that add this https://github.com/huggingface/text-generation-inference/blob//v3.1.1/router/src/server.rs#L2423 makes it output correctly. i rebuilt the code to avoid adding the input to the response.

jrc2139 · 2025-03-07T00:52:19Z

here's the code i used: https://github.com/jrc2139/text-generation-inference/blob/fix/madlad-return-full-text/router/src/server.rs

opened a draft PR to hopefully get some insight. im not so familiar with rust

#3079

github-actions bot added the Stale label Feb 11, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 18, 2024

jrc2139 mentioned this issue Mar 7, 2025

wip: comment out prepend full_text #3079

Draft

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Madlad400 support #1416

Madlad400 support #1416

irshadbhat commented Jan 9, 2024 •

edited

Loading

Narsil commented Jan 10, 2024

irshadbhat commented Jan 11, 2024

github-actions bot commented Feb 11, 2024

jhargraveiii commented Jan 16, 2025

jhargraveiii commented Jan 16, 2025

josh-at-straker commented Jan 19, 2025

jrc2139 commented Mar 6, 2025 •

edited

Loading

jrc2139 commented Mar 7, 2025 •

edited

Loading

Madlad400 support #1416

Madlad400 support #1416

Comments

irshadbhat commented Jan 9, 2024 • edited Loading

Model description

Open source status

Provide useful links for the implementation

System Info

Information

Tasks

Reproduction

Expected behavior

Narsil commented Jan 10, 2024

irshadbhat commented Jan 11, 2024

github-actions bot commented Feb 11, 2024

jhargraveiii commented Jan 16, 2025

jhargraveiii commented Jan 16, 2025

josh-at-straker commented Jan 19, 2025

jrc2139 commented Mar 6, 2025 • edited Loading

jrc2139 commented Mar 7, 2025 • edited Loading

irshadbhat commented Jan 9, 2024 •

edited

Loading

jrc2139 commented Mar 6, 2025 •

edited

Loading

jrc2139 commented Mar 7, 2025 •

edited

Loading