[Feature] Make PPO ready for text-based data

[vmoens]

Stack from ghstack (oldest at bottom):

• -> [Feature] Make PPO ready for text-based data #2855
• [BugFix] Right log-prob size in transformer wrapper #2854
• [Feature] Tokenizer for LLMEnv #2852

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2855

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 17 New Failures, 2 Pending, 2 Unrelated Failures

As of commit 4122dfc with merge base 619fec6 ():

NEW FAILURES - The following jobs have failed:

• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cpu_
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda12_6 (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu126_
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda12_8 (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu128_
• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Generate documentation / build-docs (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 9228d6cae98890a96a6407c0573012569533356e48ffe8effc3948a8503ac4e8 /exec failed with exit code 2
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t b137f938cb4834d03562c8b623350e979b3ec018e798d1b04ba4bdd2ccd37d17 /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t bf1d40acc36121d284a380256cf3c88b0b2f15bfdf59bd722152d28351b0e255 /exec failed with exit code 2
• Lint / python-source-and-configs / linux-job (gh)
  RuntimeError: Command docker exec -t 3a4f1f26cf65de0f94ea9f07deeb44fa91bbfd63a012fa98e0eec5e77b780e37 /exec failed with exit code 1
• LLM Tests on Linux / unittests (3.9, 12.8) / linux-job (gh)
  test/test_actors.py::TestLLMActor::test_from_vllm_logprobs[False-tokens2-None]
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t eac6bc21cdb7bfd0ab87f77e78e4f7fba488228cc6754035fddda7a2df14c08e /exec failed with exit code 1
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_transforms.py::TestKLRewardTransform::test_kl_diff[False]
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_transforms.py::TestKLRewardTransform::test_kl_diff[False]
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_transforms.py::TestKLRewardTransform::test_kl_diff[False]
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  RuntimeError: Command docker exec -t 8c54873ca2c5ccf84dcc5403e87b112d21ccc5bd4e3f033cc802ff67fdc28fef /exec failed with exit code 2
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t 2396133068fde81408cf00f4ebe4af15324d13e3031e225d89098214e9c5504a /exec failed with exit code 1
• Unit-tests on Linux / tests-optdeps (3.11, 12.8) / linux-job (gh)
  test/test_transforms.py::TestKLRewardTransform::test_kl_diff[False]

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda11_8 (gh) (trunk failure)
• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh) (trunk failure)
  test/test_transforms.py::TestTimer::test_transform_env

This comment was automatically generated by Dr. CI and updates every 15 minutes.