Gaudi: clean cuda/rocm code in hpu backend, enable flat_hpu

[sywangyi]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[baptistecolle]

Hey @sywangyi Thanks for contributing to the new upstreamed Gaudi backend 🙌

Here are some guidelines about the new backend that I think could be useful to you.

Guidelines for contributing to Gaudi on TGI: All changes should be made within the backends/gaudi folder. In general, you should avoid modifying the router, launcher, or benchmark to accommodate Gaudi hardware, as all Gaudi-specific logic should be contained within the backends/gaudi folder.

text-generation-inference/docs/source/backends/gaudi.mdx

Line 294 in 48df618

**Guidelines for contributing to Gaudi on TGI:** All changes should be made within the `backends/gaudi` folder. In general, you should avoid modifying the router, launcher, or benchmark to accommodate Gaudi hardware, as all Gaudi-specific logic should be contained within the `backends/gaudi` folder.

[baptistecolle]

FYI @regisss

When modifying anything outside of strictly Gaudi-related files (e.g., backend/gaudi, Dockerfile_gaudi, or Gaudi documentation), we should also tag @Narsil for review. This helps ensure that we don’t unintentionally modify or bloat core TGI components and allows the core maintainers to review the changes.

[baptistecolle]

Could you please rebase your image on the latest main branch of TGI?
Commit: 8c2c348f3c4d1101e862af493b44e4986119b557

This update integrates the latest changes from the old tgi-gaudi fork, allowing us to confirm that your refactoring is fully compatible with the latest version of the Gaudi backend.

Thanks!

[baptistecolle]

Thanks for the huge effort!

I’ve added @regisss and @Narsil as reviewers. @Narsil, could you check the modifications to the launcher and router and confirm that everything aligns with the main TGI team? Since this is a big PR, it would also be great for @regisss to take a look.

Really appreciate the refactoring and code cleanup!

One quick question—did you happen to benchmark the performance after this refactoring? Specifically, I’m curious about the impact of using the vllm-hpu-extension

[sywangyi]

Thanks for the huge effort!

I’ve added @regisss and @Narsil as reviewers. @Narsil, could you check the modifications to the launcher and router and confirm that everything aligns with the main TGI team? Since this is a big PR, it would also be great for @regisss to take a look.

Really appreciate the refactoring and code cleanup!

One quick question—did you happen to benchmark the performance after this refactoring? Specifically, I’m curious about the impact of using the vllm-hpu-extension

This PR is for functional enabling, I am also working in perf optimization. because we have comparison between tgi-gaudi(OH backend) and vllm-gaudi(hpu page path). vllm-gaudi is better than tgi-gaudi in throughput. this refactor is help to align the performance with vllm gaudi

[sywangyi]

@baptistecolle I have rebase the PR to main. please help review it

[sywangyi]

@regisss have rebase the PR to main. please help review it and merge

[regisss]

LGTM!

Gently pinging @Narsil for final approval.