menory leak

[JackHuang0701]

Hi @retsuh-bqw ,
I’m currently pretraining the Lam at stage1, and I’ve run into a persistent memory leak issue: the RAM usage keeps increasing steadily during training, eventually leading to an OOM error.

I’d like to ask:
What shuffle_buffer_size do you recommend for 8 A100 machines during stage1 pretraining? Did you encounter similar memory issues during your experiments?

[retsuh-bqw]

As mentioned in the config file, our default setting is optimized for a system with 1,600GB of memory. If your setup has different specifications, you can adjust the values proportionally.

[JackHuang0701]

8 A100 with RAM 1600 GB？

[JackHuang0701]

if training LAM on a less diverse, single dataset (e.g., BridgeV2 only)，what's the success rate? @retsuh-bqw

[retsuh-bqw]

8 A100 with RAM 1600 GB？

Yes