When training a neural network with keras for the categorical_crossentropy loss, how exactly is the loss defined? I expect it to be the average over all samples of $$\textstyle\text{loss}(p^\text{true}, p^\text{predict}) = -\sum_i p_i^\text{true} \log p_i^\text{predict}$$ but couldn't find a definitive answer in the docs nor in the code. An authoritative reference is desirable.
Looking at the code I'm not sure if the computation is delegated to tensorflow/theano.
(There is an analogous question concerning the accuracy; the code is clearer, but I don't see a call to mean()?)
PS. From this code, it appears that loss and accuracy are computed as in loss_and_acc(...) but before the last training epoch (keras version 2.0.4, same results for tensorflow and theano backend).
#!/usr/bin/python3 import numpy as np from numpy.random import randint, seed from keras import __version__ as keras_version from keras.models import Sequential from keras.layers import Dense N = 4 # Classes S = 10 # Samples nn = Sequential() nn.add(Dense(input_dim=1, units=N, kernel_initializer='normal', activation='softmax')) nn.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) seed(7) X = np.random.random((S, 1)) Y = np.vstack([np.eye(1, N, k=randint(0, N)) for _ in range(S)]) #for (x, y) in zip(X, Y) : print(x, y) def loss_and_acc(NN, X, Y) : loss = [] acc = [] for (p, q) in zip(Y, NN.predict(X)) : loss += [ -sum(a*np.log(b) for (a, b) in zip(p, q) if (b != 0)) ] acc += [ np.argmax(p) == np.argmax(q) ] return (np.mean(loss), np.mean(acc)) print("Keras version: ", keras_version) for _ in range(10) : print("Before: loss = {}, acc = {}".format(*loss_and_acc(nn, X, Y))) H = nn.fit(X, Y, epochs=1, verbose=0).history print("History: loss = {}, acc = {}".format(H['loss'][-1], H['acc'][-1])) Output:
Using Theano backend. Keras version: 2.0.4 Before: loss = 1.3843669414520263, acc = 0.2 History: loss = 1.3843669891357422, acc = 0.20000000298023224 Before: loss = 1.3834303855895995, acc = 0.2 History: loss = 1.3834303617477417, acc = 0.20000000298023224 Before: loss = 1.3824962615966796, acc = 0.3 History: loss = 1.3824962377548218, acc = 0.30000001192092896 Before: loss = 1.381564486026764, acc = 0.3 History: loss = 1.3815644979476929, acc = 0.30000001192092896 Before: loss = 1.380635154247284, acc = 0.3 History: loss = 1.380635142326355, acc = 0.30000001192092896 Before: loss = 1.3797082901000977, acc = 0.3 History: loss = 1.3797082901000977, acc = 0.30000001192092896 Before: loss = 1.378783941268921, acc = 0.2 History: loss = 1.378783941268921, acc = 0.20000000298023224 Before: loss = 1.3778621554374695, acc = 0.2 History: loss = 1.3778622150421143, acc = 0.20000000298023224 Before: loss = 1.3769428968429565, acc = 0.2 History: loss = 1.3769428730010986, acc = 0.20000000298023224 Before: loss = 1.3760262489318849, acc = 0.3 History: loss = 1.3760262727737427, acc = 0.30000001192092896 
