0
$\begingroup$

I am training a CNN on some new dataset.

Usually, the accuracy steadily improves over 10-20 epochs.

I have created a new but similar dataset (using same methods) but now I see a sharp drop after 7th epoch, from which is never recovers.

What might this mean I am doing wrong?

3520/3662 [===========================>..] - ETA: 7s - loss: 0.1260 - acc: 0.9753 3552/3662 [============================>.] - ETA: 5s - loss: 0.1294 - acc: 0.9752 3584/3662 [============================>.] - ETA: 4s - loss: 0.1283 - acc: 0.9754 3616/3662 [============================>.] - ETA: 2s - loss: 0.1360 - acc: 0.9751 3648/3662 [============================>.] - ETA: 0s - loss: 0.1348 - acc: 0.9753 3662/3662 [==============================] - 199s 54ms/step - loss: 0.1387 - acc: 0.9752 Epoch 7/50 32/3662 [..............................] - ETA: 3:29 - loss: 1.1921e-07 - acc: 1.0000 64/3662 [..............................] - ETA: 3:44 - loss: 1.1921e-07 - acc: 1.0000 96/3662 [..............................] - ETA: 3:43 - loss: 0.5037 - acc: 0.9688 128/3662 [>.............................] - ETA: 3:36 - loss: 0.6296 - acc: 0.9609 160/3662 [>.............................] - ETA: 3:29 - loss: 0.8059 - acc: 0.9500 192/3662 [>.............................] - ETA: 3:23 - loss: 0.7555 - acc: 0.9531 224/3662 [>.............................] - ETA: 3:20 - loss: 0.7915 - acc: 0.9509 256/3662 [=>............................] - ETA: 3:17 - loss: 0.6926 - acc: 0.9570 288/3662 [=>............................] - ETA: 3:14 - loss: 0.7276 - acc: 0.9549 320/3662 [=>............................] - ETA: 3:11 - loss: 0.9570 - acc: 0.9406 352/3662 [=>............................] - ETA: 3:08 - loss: 1.0990 - acc: 0.9318 384/3662 [==>...........................] - ETA: 3:05 - loss: 1.1531 - acc: 0.9271 416/3662 [==>...........................] - ETA: 3:02 - loss: 1.1032 - acc: 0.9303 448/3662 [==>...........................] - ETA: 3:00 - loss: 1.2832 - acc: 0.9174 480/3662 [==>...........................] - ETA: 2:58 - loss: 1.3991 - acc: 0.9104 512/3662 [===>..........................] - ETA: 2:55 - loss: 1.4691 - acc: 0.9062 544/3662 [===>..........................] - ETA: 2:53 - loss: 1.4420 - acc: 0.9081 576/3662 [===>..........................] - ETA: 2:50 - loss: 1.5018 - acc: 0.9045 608/3662 [===>..........................] - ETA: 2:48 - loss: 1.4492 - acc: 0.9079 640/3662 [====>.........................] - ETA: 2:46 - loss: 1.3768 - acc: 0.9125 672/3662 [====>.........................] - ETA: 2:44 - loss: 1.3112 - acc: 0.9167 704/3662 [====>.........................] - ETA: 2:42 - loss: 1.3431 - acc: 0.9148 736/3662 [=====>........................] - ETA: 2:40 - loss: 1.3066 - acc: 0.9171 768/3662 [=====>........................] - ETA: 2:38 - loss: 1.2941 - acc: 0.9180 800/3662 [=====>........................] - ETA: 2:36 - loss: 1.3431 - acc: 0.9150 832/3662 [=====>........................] - ETA: 2:34 - loss: 1.4464 - acc: 0.9087 864/3662 [======>.......................] - ETA: 2:32 - loss: 1.5980 - acc: 0.8993 896/3662 [======>.......................] - ETA: 2:30 - loss: 1.7568 - acc: 0.8895 928/3662 [======>.......................] - ETA: 2:28 - loss: 2.0263 - acc: 0.8728 960/3662 [======>.......................] - ETA: 2:27 - loss: 2.1434 - acc: 0.8656 992/3662 [=======>......................] - ETA: 2:25 - loss: 2.3017 - acc: 0.8558 1024/3662 [=======>......................] - ETA: 2:23 - loss: 2.3557 - acc: 0.8525 1056/3662 [=======>......................] - ETA: 2:21 - loss: 2.5133 - acc: 0.8428 1088/3662 [=======>......................] - ETA: 2:20 - loss: 2.6468 - acc: 0.8346 1120/3662 [========>.....................] - ETA: 2:18 - loss: 2.8590 - acc: 0.8214 1152/3662 [========>.....................] - ETA: 2:16 - loss: 2.9475 - acc: 0.8160 1184/3662 [========>.....................] - ETA: 2:14 - loss: 3.1265 - acc: 0.8049 1216/3662 [========>.....................] - ETA: 2:13 - loss: 3.2430 - acc: 0.7977 1248/3662 [=========>....................] - ETA: 2:11 - loss: 3.3794 - acc: 0.7893 1280/3662 [=========>....................] - ETA: 2:09 - loss: 3.4964 - acc: 0.7820 1312/3662 [=========>....................] - ETA: 2:07 - loss: 3.5831 - acc: 0.7767 1344/3662 [==========>...................] - ETA: 2:05 - loss: 3.7257 - acc: 0.7679 1376/3662 [==========>...................] - ETA: 2:04 - loss: 3.8030 - acc: 0.7631 1408/3662 [==========>...................] - ETA: 2:02 - loss: 3.8883 - acc: 0.7578 1440/3662 [==========>...................] - ETA: 2:00 - loss: 4.0369 - acc: 0.7486 1472/3662 [===========>..................] - ETA: 1:58 - loss: 4.0587 - acc: 0.7473 1504/3662 [===========>..................] - ETA: 1:56 - loss: 4.2081 - acc: 0.7380 1536/3662 [===========>..................] - ETA: 1:55 - loss: 4.2673 - acc: 0.7344 1568/3662 [===========>..................] - ETA: 1:53 - loss: 4.4167 - acc: 0.7251 1600/3662 [============>.................] - ETA: 1:51 - loss: 4.4895 - acc: 0.7206 1632/3662 [============>.................] - ETA: 1:49 - loss: 4.5694 - acc: 0.7157 1664/3662 [============>.................] - ETA: 1:48 - loss: 4.6074 - acc: 0.7133 1696/3662 [============>.................] - ETA: 1:46 - loss: 4.6916 - acc: 0.7081 1728/3662 [=============>................] - ETA: 1:44 - loss: 4.7726 - acc: 0.7031 1760/3662 [=============>................] - ETA: 1:42 - loss: 4.8415 - acc: 0.6989 1792/3662 [=============>................] - ETA: 1:40 - loss: 4.8810 - acc: 0.6964 1824/3662 [=============>................] - ETA: 1:39 - loss: 4.9279 - acc: 0.6935 1856/3662 [==============>...............] - ETA: 1:37 - loss: 4.9558 - acc: 0.6918 1888/3662 [==============>...............] - ETA: 1:35 - loss: 4.9828 - acc: 0.6901 1920/3662 [==============>...............] - ETA: 1:33 - loss: 4.9921 - acc: 0.6896 1952/3662 [==============>...............] - ETA: 1:32 - loss: 4.9846 - acc: 0.6901 1984/3662 [===============>..............] - ETA: 1:30 - loss: 5.0423 - acc: 0.6865 2016/3662 [===============>..............] - ETA: 1:28 - loss: 5.1062 - acc: 0.6825 2048/3662 [===============>..............] - ETA: 1:26 - loss: 5.1602 - acc: 0.6792 2080/3662 [================>.............] - ETA: 1:25 - loss: 5.1970 - acc: 0.6769 2112/3662 [================>.............] - ETA: 1:23 - loss: 5.2022 - acc: 0.6766 2144/3662 [================>.............] - ETA: 1:21 - loss: 5.2449 - acc: 0.6740 2176/3662 [================>.............] - ETA: 1:20 - loss: 5.2937 - acc: 0.6710 2208/3662 [=================>............] - ETA: 1:18 - loss: 5.3337 - acc: 0.6685 2240/3662 [=================>............] - ETA: 1:16 - loss: 5.3295 - acc: 0.6687 2272/3662 [=================>............] - ETA: 1:14 - loss: 5.3538 - acc: 0.6673 2304/3662 [=================>............] - ETA: 1:13 - loss: 5.3843 - acc: 0.6654 2336/3662 [==================>...........] - ETA: 1:11 - loss: 5.4072 - acc: 0.6640 2368/3662 [==================>...........] - ETA: 1:09 - loss: 5.4022 - acc: 0.6643 2400/3662 [==================>...........] - ETA: 1:08 - loss: 5.4510 - acc: 0.6613 2432/3662 [==================>...........] - ETA: 1:06 - loss: 5.4853 - acc: 0.6591 2464/3662 [===================>..........] - ETA: 1:04 - loss: 5.5188 - acc: 0.6571 2496/3662 [===================>..........] - ETA: 1:02 - loss: 5.5707 - acc: 0.6538 2528/3662 [===================>..........] - ETA: 1:01 - loss: 5.5958 - acc: 0.6523 2560/3662 [===================>..........] - ETA: 59s - loss: 5.6455 - acc: 0.6492 2592/3662 [====================>.........] - ETA: 57s - loss: 5.6691 - acc: 0.6478 2624/3662 [====================>.........] - ETA: 55s - loss: 5.7044 - acc: 0.6456 2656/3662 [====================>.........] - ETA: 54s - loss: 5.7631 - acc: 0.6419 2688/3662 [=====================>........] - ETA: 52s - loss: 5.7964 - acc: 0.6399 2720/3662 [=====================>........] - ETA: 50s - loss: 5.8467 - acc: 0.6368 2752/3662 [=====================>........] - ETA: 48s - loss: 5.8783 - acc: 0.6348 2784/3662 [=====================>........] - ETA: 47s - loss: 5.8918 - acc: 0.6340 2816/3662 [======================>.......] - ETA: 45s - loss: 5.9279 - acc: 0.6317 2848/3662 [======================>.......] - ETA: 43s - loss: 5.9405 - acc: 0.6310 2880/3662 [======================>.......] - ETA: 42s - loss: 5.9640 - acc: 0.6295 2912/3662 [======================>.......] - ETA: 40s - loss: 6.0037 - acc: 0.6271 2944/3662 [=======================>......] - ETA: 38s - loss: 6.0260 - acc: 0.6257 2976/3662 [=======================>......] - ETA: 36s - loss: 6.0370 - acc: 0.6250 3008/3662 [=======================>......] - ETA: 35s - loss: 6.0639 - acc: 0.6233 3040/3662 [=======================>......] - ETA: 33s - loss: 6.0743 - acc: 0.6227 3072/3662 [========================>.....] - ETA: 31s - loss: 6.0897 - acc: 0.6217 3104/3662 [========================>.....] - ETA: 29s - loss: 6.1048 - acc: 0.6208 3136/3662 [========================>.....] - ETA: 28s - loss: 6.1351 - acc: 0.6189 3168/3662 [========================>.....] - ETA: 26s - loss: 6.1443 - acc: 0.6184 3200/3662 [=========================>....] - ETA: 24s - loss: 6.1685 - acc: 0.6169 3232/3662 [=========================>....] - ETA: 23s - loss: 6.1822 - acc: 0.6160 3264/3662 [=========================>....] - ETA: 21s - loss: 6.1858 - acc: 0.6158 3296/3662 [==========================>...] - ETA: 19s - loss: 6.1991 - acc: 0.6150 3328/3662 [==========================>...] - ETA: 17s - loss: 6.2073 - acc: 0.6145 3360/3662 [==========================>...] - ETA: 16s - loss: 6.2729 - acc: 0.6104 3392/3662 [==========================>...] - ETA: 14s - loss: 6.2945 - acc: 0.6091 3424/3662 [===========================>..] - ETA: 12s - loss: 6.2875 - acc: 0.6095 3456/3662 [===========================>..] - ETA: 11s - loss: 6.3225 - acc: 0.6073 3488/3662 [===========================>..] - ETA: 9s - loss: 6.3431 - acc: 0.6061 3520/3662 [===========================>..] - ETA: 7s - loss: 6.3633 - acc: 0.6048 3552/3662 [============================>.] - ETA: 5s - loss: 6.4058 - acc: 0.6022 3584/3662 [============================>.] - ETA: 4s - loss: 6.3980 - acc: 0.6027 3616/3662 [============================>.] - ETA: 2s - loss: 6.4217 - acc: 0.6012 3648/3662 [============================>.] - ETA: 0s - loss: 6.4537 - acc: 0.5992 3662/3662 [==============================] - 200s 55ms/step - loss: 6.4730 - acc: 0.5980 Epoch 8/50
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't know anything about the data you're using so I'll offer some suggestions.

Your learning rate might be too high. Maybe you're over-shooting the gradient descent such that after you approach a nice vector of coefficients, you're then jumping too far in one epoch and thus losing progress towards the "global optima".

Your activation function might be saturating your units.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am using a standard "un-trained" AlexNet CNN, and learning rate etc are the default Keras values. How can I check if I am saturating my units? $\endgroup$ Commented Apr 23, 2019 at 13:57
  • $\begingroup$ You can edit the default values. Lower the learning rate and try some of these activation functions: keras.io/layers/advanced-activations $\endgroup$ Commented Apr 23, 2019 at 14:07
  • $\begingroup$ You were right about the learning rate, that was the issue. Strange how default lr=0.01 was too fast. $\endgroup$ Commented Apr 26, 2019 at 15:41
  • $\begingroup$ Glad to hear it's $\endgroup$ Commented Apr 26, 2019 at 16:22

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.