1
$\begingroup$

I have a multiclass classification data where the target has 11 classes. I am trying to build a Neural Net using Keras. I am using softmax as activation function and categorical_crossentropy as the loss function. I have one hot encoded the target before passing it into the net. The issue I am facing is which Keras metric should I use for this purpose? The official documentation does not mention which metric is suitable for multiclass classification.

This link mentions to use categorical_accuracy as the metric for multiclass classification but other than that, all other question on this site are about multilabel classification metrics like this and this link.

Is there any implementation of lets say f1_score in Keras using the custom metric function, since f1_score is the go to metric for multiclass classification I guess?

EDIT 1:

Would something like this work using the custom metric functionality in Keras?

from sklearn.metrics import f1_score def my_metric_fn(y_true, y_pred): f1 = f1_score(y_true, y_pred) return f1 model = Sequential() model.add(Dense(input_dim = 12, units = 128, activation = 'relu', kernel_initializer = 'he_uniform')) model.add(Dense(units = 64, activation = 'relu', kernel_initializer = 'he_uniform')) model.add(Dense(units = 32, activation = 'relu', kernel_initializer = 'he_uniform')) model.add(Dense(units = 11, activation = 'softmax', kernel_initializer = 'glorot_uniform')) model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = [my_metric_fn]) history = model.fit(train_x5, train_encoded, validation_split = 0.2, epochs = 5, batch_size = 1000)

EDIT 2: This is giving me an error in the last line as follows: OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

Improve this question
$\endgroup$
6
  • $\begingroup$ For multiclass classification you can simply use a categorical cross entropy loss function. Depending on whether or not the values are one-hot encoded you would have to use either the sparse categorical cross entropy loss or the normal categorical cross entropy loss. $\endgroup$ Commented Dec 14, 2021 at 12:58
  • $\begingroup$ I am asking about the metric to use not the loss. $\endgroup$ Commented Dec 14, 2021 at 13:43
  • $\begingroup$ It would help if you said what you would like the metric to evaluate and why it should be different from the loss function. $\endgroup$ Commented Dec 14, 2021 at 14:53
  • $\begingroup$ The metric needs to be any metric that is used in multiclass classification like f1_score or kappa. But Keras has not yet implemented them yet unlike sklearn. $\endgroup$ Commented Dec 14, 2021 at 15:37
  • $\begingroup$ it should be different from the loss function. I find this statement interesting as it implies that it is not necessary to use metrics to evaluate the model. Does this mean we can just use the loss function for the evaluation and not the metrics? $\endgroup$ Commented Dec 14, 2021 at 15:38

1 Answer 1

Reset to default
3
$\begingroup$

One option is to implement F1 score in Keras:

from tensorflow.keras import backend as K def f1(y_true, y_pred): def recall_m(y_true, y_pred): TP = K.sum(K.round(K.clip(y_true * y_pred, 0, 1))) Positives = K.sum(K.round(K.clip(y_true, 0, 1))) recall = TP / (Positives+K.epsilon()) return recall def precision_m(y_true, y_pred): TP = K.sum(K.round(K.clip(y_true * y_pred, 0, 1))) Pred_Positives = K.sum(K.round(K.clip(y_pred, 0, 1))) precision = TP / (Pred_Positives+K.epsilon()) return precision precision, recall = precision_m(y_true, y_pred), recall_m(y_true, y_pred) return 2*((precision*recall)/(precision+recall+K.epsilon()))
Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I would need to call the function in the model.compile where I would need y_pred which I can only get after I fit and evaluate the model. But fitting and evaluating comes after compiling so how can I call the function in model.compile? Even if I pass the function as a list (as I have done in EDIT 1), I will still get the same error as in EDIT 2! $\endgroup$ Commented Dec 17, 2021 at 12:11
  • 2
    $\begingroup$ Ok I implemented it in my model and its works! NVM my previous comment!XD $\endgroup$ Commented Dec 23, 2021 at 12:52
  • 1
    $\begingroup$ You could also use the built in method here tensorflow.org/api_docs/python/tf/keras/metrics/F1Score $\endgroup$ Commented Nov 30, 2023 at 21:40

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.