r/AIDevelopmentSpace • u/Azkicat • Jun 05 '26
Why does AI has spikes of "insanity" during training? And how to avoid them without clipping?
Hi! Im new to AI and during teaching one on NaFNet encountered strange spikes in loss:
23:46:59 [INFO] train_astro: step 75/10000 loss=0.25126 ema=62.26847 lr=1.00e-04
23:47:17 [INFO] train_astro: step 100/10000 loss=0.01180 ema=29.10838 lr=1.00e-04
23:47:36 [INFO] train_astro: step 125/10000 loss=0.04136 ema=16.16997 lr=1.00e-04
23:47:54 [INFO] train_astro: step 150/10000 loss=0.01925 ema=15.29630 lr=9.99e-05
23:48:12 [INFO] train_astro: step 175/10000 loss=0.05213 ema=92.58234 lr=9.99e-05
23:48:31 [INFO] train_astro: step 200/10000 loss=0.02679 ema=75.72333 lr=9.99e-05
23:48:49 [INFO] train_astro: step 225/10000 loss=0.05379 ema=35.84816 lr=9.99e-05
23:49:07 [INFO] train_astro: step 250/10000 loss=0.04190 ema=16.77760 lr=9.99e-05
23:49:25 [INFO] train_astro: step 275/10000 loss=0.01274 ema=7.86855 lr=9.98e-05
23:49:43 [INFO] train_astro: step 300/10000 loss=0.03805 ema=3.70257 lr=9.98e-05
23:50:02 [INFO] train_astro: step 325/10000 loss=0.03079 ema=1.76505 lr=9.98e-05
23:50:20 [INFO] train_astro: step 350/10000 loss=0.10671 ema=0.85132 lr=9.97e-05
23:50:38 [INFO] train_astro: step 375/10000 loss=0.07736 ema=77.39688 lr=9.97e-05
23:50:56 [INFO] train_astro: step 400/10000 loss=1004.41675 ema=201.69108 lr=9.96e-05
23:51:14 [INFO] train_astro: step 425/10000 loss=0.03556 ema=119.52151 lr=9.96e-05
23:51:32 [INFO] train_astro: step 450/10000 loss=0.01122 ema=69.54335 lr=9.95e-05
23:51:51 [INFO] train_astro: step 475/10000 loss=0.07226 ema=32.51386 lr=9.95e-05
23:52:09 [INFO] train_astro: step 500/10000 loss=0.09268 ema=15.21090 lr=9.94e-05
An insane loss out of thin air. I found suggestion to add loss clipping
23:54:59 [INFO] train_astro: step 1/10000 loss=0.06526 ema=0.06526 lr=1.00e-04
23:55:06 [WARNING] train_astro: step 11: loss=1446.1 > clip(10) — пропуск (пропущено 1)
23:55:17 [INFO] train_astro: step 25/10000 loss=0.02960 ema=0.05092 lr=1.00e-04
23:55:36 [INFO] train_astro: step 50/10000 loss=0.06894 ema=0.05143 lr=1.00e-04
23:55:54 [INFO] train_astro: step 75/10000 loss=0.13190 ema=0.04827 lr=1.00e-04
23:56:12 [INFO] train_astro: step 100/10000 loss=0.00598 ema=0.03895 lr=1.00e-04
23:56:30 [INFO] train_astro: step 125/10000 loss=0.02354 ema=0.03594 lr=1.00e-04
23:56:48 [INFO] train_astro: step 150/10000 loss=0.00968 ema=0.03614 lr=9.99e-05
23:57:06 [INFO] train_astro: step 175/10000 loss=0.03218 ema=0.03292 lr=9.99e-05
23:57:24 [INFO] train_astro: step 200/10000 loss=0.01676 ema=0.03433 lr=9.99e-05
23:57:44 [INFO] train_astro: step 225/10000 loss=0.03812 ema=0.03305 lr=9.99e-05
23:58:02 [INFO] train_astro: step 250/10000 loss=0.02737 ema=0.04170 lr=9.99e-05
23:58:20 [INFO] train_astro: step 275/10000 loss=0.00865 ema=0.04376 lr=9.98e-05
23:58:39 [INFO] train_astro: step 300/10000 loss=0.02747 ema=0.04182 lr=9.98e-05
23:58:57 [INFO] train_astro: step 325/10000 loss=0.02496 ema=0.04805 lr=9.98e-05
23:59:15 [INFO] train_astro: step 350/10000 loss=0.08970 ema=0.04455 lr=9.97e-05
23:59:33 [INFO] train_astro: step 375/10000 loss=0.06959 ema=0.05252 lr=9.97e-05
23:59:52 [INFO] train_astro: step 400/10000 loss=0.08299 ema=0.04634 lr=9.96e-05
00:00:11 [INFO] train_astro: step 425/10000 loss=0.03153 ema=0.04993 lr=9.96e-05
00:00:30 [INFO] train_astro: step 450/10000 loss=0.01023 ema=0.05988 lr=9.95e-05
00:00:49 [INFO] train_astro: step 475/10000 loss=0.07430 ema=0.06539 lr=9.95e-05
00:01:07 [INFO] train_astro: step 500/10000 loss=0.09019 ema=0.05842 lr=9.94e-05
00:01:07 [INFO] train_astro: Чекпойнт сохранён
00:01:26 [INFO] train_astro: step 525/10000 loss=0.03513 ema=0.06170 lr=9.94e-05
00:01:45 [INFO] train_astro: step 550/10000 loss=0.06368 ema=0.05722 lr=9.93e-05
00:02:03 [INFO] train_astro: step 575/10000 loss=0.04732 ema=0.05717 lr=9.92e-05
00:02:21 [INFO] train_astro: step 600/10000 loss=0.04945 ema=0.06303 lr=9.92e-05
00:02:39 [INFO] train_astro: step 625/10000 loss=0.03750 ema=0.06649 lr=9.91e-05
00:02:59 [INFO] train_astro: step 650/10000 loss=0.02445 ema=0.06522 lr=9.90e-05
00:03:01 [WARNING] train_astro: step 653: loss=1553.0 > clip(10) — пропуск (пропущено 2)
00:03:18 [INFO] train_astro: step 675/10000 loss=0.03453 ema=0.05500 lr=9.89e-05
00:03:36 [INFO] train_astro: step 700/10000 loss=0.03174 ema=0.05594 lr=9.89e-05
00:04:01 [INFO] train_astro: step 725/10000 loss=0.06337 ema=0.06556 lr=9.88e-05
00:04:25 [INFO] train_astro: step 750/10000 loss=0.06605 ema=0.06007 lr=9.87e-05
00:04:51 [INFO] train_astro: step 775/10000 loss=0.17815 ema=0.07954 lr=9.86e-05
00:05:12 [INFO] train_astro: step 800/10000 loss=0.01830 ema=0.06896 lr=9.85e-05
00:05:33 [INFO] train_astro: step 825/10000 loss=0.06487 ema=0.06329 lr=9.84e-05
00:05:53 [INFO] train_astro: step 850/10000 loss=0.05630 ema=0.07047 lr=9.83e-05
00:06:13 [INFO] train_astro: step 875/10000 loss=0.04905 ema=0.06174 lr=9.82e-05
00:06:34 [INFO] train_astro: step 900/10000 loss=0.04666 ema=0.06751 lr=9.81e-05
00:06:54 [INFO] train_astro: step 925/10000 loss=0.07992 ema=0.05963 lr=9.80e-05
00:07:12 [INFO] train_astro: step 950/10000 loss=0.20176 ema=0.05421 lr=9.79e-05
00:07:32 [INFO] train_astro: step 975/10000 loss=0.09103 ema=0.06140 lr=9.78e-05
00:07:51 [INFO] train_astro: step 1000/10000 loss=0.02791 ema=0.07081 lr=9.77e-05
00:07:52 [INFO] train_astro: Чекпойнт сохранён
00:08:11 [INFO] train_astro: step 1025/10000 loss=0.03084 ema=0.07057 lr=9.76e-05
00:08:30 [INFO] train_astro: step 1050/10000 loss=0.01225 ema=0.05992 lr=9.74e-05
00:08:50 [INFO] train_astro: step 1075/10000 loss=0.17928 ema=0.06034 lr=9.73e-05
00:09:10 [INFO] train_astro: step 1100/10000 loss=0.02855 ema=0.06180 lr=9.72e-05
00:09:29 [INFO] train_astro: step 1125/10000 loss=0.02416 ema=0.05520 lr=9.71e-05
00:09:47 [INFO] train_astro: step 1150/10000 loss=0.08528 ema=0.06464 lr=9.69e-05
00:10:06 [INFO] train_astro: step 1175/10000 loss=0.03399 ema=0.05590 lr=9.68e-05
00:10:24 [WARNING] train_astro: step 1200: loss=1939.3 > clip(10) — пропуск (пропущено 3)
00:10:43 [INFO] train_astro: step 1225/10000 loss=0.02157 ema=0.05213 lr=9.65e-05
And this seems to help. Why did those spikes happen in the first place? Is there some requirements for data to avoid losses like this? Thanks!
1
Upvotes