I am using a CNN to classify medical images. I am using a four convolutional layers with ReLU activation followed by a softmax layer. I am using rmsprop as the optimizer. The problem I am facing is the network's test accuracy increases and then goes down. This continues till the maximum number of epochs is reached. I read somewhere that this could be due to the fact that the network shifts its weights in favor of one class, and then has to do so in favor of the other class. I shuffled the test data, to see if this would help, but no such luck. Is there any remedy?