Yes, it is called "least squares" because it solves the problem of minimizing the sum of the squares of the errors. Geometrically, as you say we are finding a "closest possible vector", and "closest" is defined in terms of the Euclidean norm, which is the square root of the sum of the squares of the coordinates.

According to https://mathshistory.st-andrews.ac.uk/Miller/mathword/m/, the term "Method of least squares" (in French: "la Méthode des moindres quarrés" goes back to Legendre in 1805). This was before the development of linear algebra in the modern sense.

Yes, it is called "least squares" because it solves the problem of minimizing the sum of the squares of the errors. Geometrically, as you say we are finding a "closest possible vector", and "closest" is defined in terms of the Euclidean norm, which is the square root of the sum of the squares of the coordinates.

According to https://mathshistory.st-andrews.ac.uk/Miller/mathword/m/, the term "Method of least squares" (in French: "la Méthode des moindres quarrés") goes back to Legendre in 1805. This was before the development of the modern machinery of linear algebra. So Legendre and Gauss would not be using vectors and matrices as such.