Currently doing a multi-label classifier. I receive suspiciously high accuracies (0.94) calling the evaluate method in keras(the evaluate method returns accuracy and loss). I then tried to program the method manually, as described here: How does Keras evaluate the accuracy?
However, I receive way lower accuracy doing it on my own. F1_mico :0.7157894736842105 Accuracy:0.3530864197530864
I do not understand why. I thought it has to do something with imbalanced classes. The f1_mico value is high, while the f1_macro is around 0.45, so it seems to make sense(I have a hierarchy, so some classes naturally appear more often than others). However, how is keras getting such a high accuracy, it does not make sense to me. The code:
model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy']) #print("Traning Model...") model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, callbacks=[checkpoint], validation_data=(X_test, y_test)) # starts training print("Testing Model") output = model.predict(X_final, batch_size = batch_size) for pred_i in output: pred_i[pred_i >=0.5] = 1 pred_i[pred_i < 0.5] = 0 print "F1: " + str(f1_score(y_final, output, average='micro')) print "Accuracy: " + str(accuracy_score(y_final, output)) mscores = model.evaluate(X_final, y_final, batch_size = batch_size) print mscores Output:
Creating Model... Testing Model F1: 0.7157894736842105 Accuracy: 0.3530864197530864 405/405 [==============================] - 8s 20ms/step ['0.15227678694106914', '0.9422222640779283'] 
