About reproducing SMART-tiny-7M

[NaNaoiSong]

Thanks for the great project! I have some questions about reproducing SMART-tiny-7M.

If I expect to reproduce the 0.7671 of SMART-tiny-7M you mentioned, I just need to run pre_bc.yaml with 32 epochs?

[zhejz]

Yes. But be aware that you need a powerful cluster node (8x A100 80GB, 240 cpus). And the training (without validation) can take about 2 days.

[NaNaoiSong]

I will give it a try. Thanks again for the refactored code! It’s much clearer and easier to understand.
Additionally, I have two more questions:

1. It seems that only "interest" and "predict" agents are considered, while "sim" agents mentioned in the official tutorial (retrieved using sim_agents.submission_specs.get_sim_agent_ids(scenario)) are not used. Why doesn’t this cause a submission failure?
2. I noticed that the all tokenizers are transferred to the model. Given the iterative process for agents, will this impact training efficiency compared to using it in the dataset?

[zhejz]

A1: The "interest" and "predict" agents are used for training (together with some other random agents). The WOSAC leaderboard submission requires the prediction of all agents which are valid at the current time step (t=10), regardless of "interest" or "predict".
A2: I guess you mean the sequential tokenization, right? Yes, for BC without data augmentation you can do it more efficiently and cache the tokenized results into the dataset. But for BC data augmentation (e.g., trajectory perturbation) or closed-loop fine-tuning, we have to do it during the training after loading a new batch. To keep our code simple and more generalizable, we do it during the training.

[OrangeSodahub]

@zhejz Hi, to reproduce the results of smart-7M and catK, I just need to change the val_k to 1 and 48 here, right?

catk/scripts/local_val.sh

Line 7 in 7fe4b80

VAL_K=48