7
$\begingroup$

I am studying how the distances between embeddings evolve during training of a language model.

One way to describe this "evolution" is that the k-nearest neighbours of a particular embedding may change after some training, and eventually "converge."

Problem

I want to compute some metric of similarity between distance distance matrices at different training steps. For example, I have a distance matrix at the beginning of training and another after some iterations, and I want to quantify how similar they are.

What I’ve tried/considered

  • Converting the upper-triangular part of the matrix into a vector and computing correlations (Pearson or Spearman) between different training steps.
  • Looking at averages and standard deviations of the distances over time, though this feels too coarse.

Question

Are there established metrics or approaches for comparing distance (or similarity) matrices in this way?

I am looking for methods that can capture whether the overall structure of the embedding space is stabilizing or converging during training.

Improve this question
$\endgroup$
2
  • 5
    $\begingroup$ Would it help to track the average difference between the matrices at each timepoint? Each time there's a new matrix, report the MAE with respect to the previous one (or MSE, max difference, etc). I would expect that as it converges, the difference between subsequent matrices becomes smaller and smaller. $\endgroup$ Commented Sep 24 at 17:05
  • 5
    $\begingroup$ You could think of the matrices as grey-scale images and use image comparision metrics like Structural Similarity Index. These indices are in some ways sensible to structure changes in the "images". Did you try the usual measures in metric spaces? There are "matrix-versions" of the common euclidean distance, related to the MSE. $\endgroup$ Commented Sep 25 at 6:59

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.