assert indexed_indices.size>0 error

[helloMr996]

When pre-training a large model, errors occur when inputting a small data set

[Advait251206]

This error is most commonly caused by the dataset being too small (or not producing any valid samples) rather than by the model itself.

An assertion like:

assert indexed_indices.size > 0

usually indicates that, after preprocessing and indexing, the dataset builder ended up with zero valid indices.

Some things to check are:

1. Dataset size

   • Very small datasets may not produce enough samples after tokenization and sequence packing, especially if your seq-length is large.

2. Sequence length

   • If your documents are all shorter than the configured --seq-length, very few (or even zero) training samples may be generated, depending on the preprocessing settings.

3. Preprocessing output

   • Verify that preprocessing completed successfully and that the generated .bin and .idx files are not empty or corrupted.

4. Dataset split

   • If you're using train/validation/test splits, make sure each split actually contains data. With very small datasets, one of the splits can end up empty.

5. Dataset path and configuration

   • Double-check that the paths passed to Megatron-LM point to the expected indexed dataset and not an empty directory or partially generated files.

If your goal is simply to test the training pipeline with a tiny dataset, you may also want to:

• reduce --seq-length,
• reduce --global-batch-size/--micro-batch-size if appropriate,
• or duplicate the dataset so that enough samples are available for indexing.

If you could share:

• the full stack trace,
• your preprocessing command,
• your training command,
• and approximately how many documents/tokens are in the dataset,

it would be easier to determine exactly why indexed_indices is ending up empty.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206