We’ve benchmarked a similar model on two machines with slightly different GPUs and seen little differences in the time per 100 iterations. In general, we’re at something like 12-14 hours to reach 200k iterations (not bad but not great).
As we get things working and consider upgrading to better GPUs and optimizing, I am curious which features others have found to optimize the time to train the model? I have some assumptions about this but would love to know if anyone has tested this explicitly or has experience with two very different GPUs in order to speak to this? Any other tips for speeding up the model /w different input parameters etc. would also be appreciated!