6
$\begingroup$

The figures below depict validation and training curves for metrics (top row, the lower the better) and losses (bottom row). The last column depicts aggregated metrics/losses from the first two columns.

Validation curves on metrics plots are flat from the first training epoch onward, despite a sufficiently large dataset and some regularization. This is where my first confusion is.

Another confusion is the loss on the second plot, which increases for validation, while the validation metric stays flat, as depicted in the figure directly above.

Qualitative results tell that the model is underfitted, converging at a local minimum despite sufficient capacity (43M parameters in total).

Could you help me understand these learning curves? It doesn't look like a plain overfitting.

enter image description here

Improve this question
$\endgroup$
2
  • $\begingroup$ A few questions to better understand. What kind of data is it? what exactly are the metric and the loss? are the metric levels obtained (strangely from the 1st epoch for the val set) any good? do you apply preprocessing to both datasets? how many samples do you have in both sets? $\endgroup$ Commented Mar 12 at 23:04
  • $\begingroup$ @rehaqds: 1) video data, the task of predicting a 6D object pose from frame t-1 to t; 2) the losses are both MSE, the metrics are MSE for 3D position (t_err) and geodesic distance for 3D orientation (r_err); ADD denotes a distance between a pointcloud transformed by predicted and gt poses; 3) metrics are not good; qualitatively, on the val set the model predicts 6D pose that is seemingly uncorrelated with observations (and very small); 4) the pipeline is correct, having ideal analytical predictions without the learnable model; 5) both sets are sufficiently large $\endgroup$ Commented Mar 13 at 16:27

1 Answer 1

Reset to default
6
$\begingroup$

My guess: either there is an error somewhere, or the features have nothing (or very little) to do with the target.

For example, you would get something similar if you try to predict the weather based on bitcoin price: since there is no link, the model can only overfit itself to the training data, causing nothing (or worse) on the validation set.

Since I assume that your task is not crazy, an error is very likely. For example, you would probably have this effect if there has been a randomization causing features and labels not to be aligned correctly. Or anything else causing the features <-> labels association to be messed up.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Thank you. I obtained perfect results by replacing the trainable model with an analytical method, which tells me that there is no pipeline error outside of the model/optimization code. But different models/optimization setups end up with similar results. Insufficient features to predict the label would be the next point to validate (although it should be conceptually sound as is) $\endgroup$ Commented Mar 13 at 16:35

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.