[Refactor] Modular Integration Test Framework with DeepSeek-v3 Support #1431
Conversation
meta-cla
bot
added
the
CLA Signed
wwwjn
changed the title
wwwjn
changed the title
wwwjn
changed the title
| parser.add_argument( | ||
| "--config_path", | ||
| default="./tests/integration_tests/base_config.toml", | ||
| help="Base config path for integration tests. This is the config that will be used as a base for all tests.", | ||
| ) |
There was a problem hiding this comment.
do we need to expose this anyway?
There was a problem hiding this comment.
I still keep this field for 2 reasons:
- We want to leave a little bit flexibility, for developers to quickly run integration tests locally with different config. For example, I could run the
features.pywith a deepseek debug model config. - A file path is needed since we run intergration tests using command line, and we need to specify
CONFIG_FILES={}
wwwjn
changed the title
wwwjn
force-pushed
the
ci-refactor
branch
2 times, most recently
from
7b17cc8 to
b8f7a7b
Compare
| @@ -4,7 +4,7 @@ | |||
|
|
|||
| #### A PyTorch native platform for training generative AI models | |||
|
|
|||
| [](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu.yaml?query=branch%3Amain) | |||
| [](https://github.com/pytorch/torchtitan/actions/workflows/integration_test_8gpu.yaml?query=branch%3Amain) | |||
There was a problem hiding this comment.
I think the badge will show 8 GPU Integration Test - Core Features and I feel it's too verbose.
Also do we want to include two badges?
e.g. 8 GPU Feature Tests and 8 GPU Model Tests
tests/integration_tests/models.py
Outdated
|
|
||
|
|
||
| @dataclass | ||
| class OverrideDefinitions: |
There was a problem hiding this comment.
any reason we can't reuse this?
There was a problem hiding this comment.
Oh yes I should. Let me refactor this PR a little bit to reuse the functions
| for model_name, test_list in test_case_dict.items(): | ||
| for test_flavor in test_list: | ||
| # Filter by test_name if specified | ||
| if args.test_name != "all" and test_flavor.test_name != args.test_name: |
There was a problem hiding this comment.
what if different model_names have the same test_name? Do you run them all?
| @@ -601,39 +599,48 @@ def run_test(test_flavor: OverrideDefinitions, full_path: str, output_dir: str): | |||
|
|
|||
|
|
|||
| def run_tests(args): | |||
There was a problem hiding this comment.
This code is repeatedly defined in every .py tests which I think is very unnecessary.
We only need to have separate .yaml files for GH actions.
We can have different files to host OverrideDefinitions.
But we don't need different files to host the same functions such as main, run_tests, etc.
Integration Tests Restructuring
Split tests into two sets:
Moved integration test files from the root directory to a dedicated
tests/integration_tests/directoryAdded a base configuration file
base_config.tomlfor integration tests, as most of the train_configs shared 90% same settingsRemove "use_for_integration_test" field in train configs. Change to by default using "debugmodel" flavor for integration tests.