This is a partial answer.
One gets a different answer depending upon the alphabetical order of the symbols used.
All that is done below is to swap one symbol for another.
Starting with the first expression.
Integrate[1/Sqrt[(x - t) (y - t)], {t, 0, y}, Assumptions -> 0 < y < x]
Log[(x + y + 2 Sqrt[x y])/(x - y)]
Swap a for x
Integrate[1/Sqrt[(a - t) (y - t)], {t, 0, y}, Assumptions -> 0 < y < a]
Log[(a + y + 2 Sqrt[a y])/(a - y)]
Swap b for y
Integrate[1/Sqrt[(a - t) (b - t)], {t, 0, b}, Assumptions -> 0 < b < a]
Log[(a + b + 2 Sqrt[a b])/(a - b)]
Now swap y for a
Integrate[1/Sqrt[(y - t) (b - t)], {t, 0, b}, Assumptions -> 0 < b < y]
2 ArcTanh[Sqrt[b/y]]
This appears to be a different answer.
Test the two expressions
Based upon Asim's result let's check if the two answers are equivalent.
FullSimplify[Log[(x + y + 2 Sqrt[x y])/(x - y)] - 2 ArcTanh[Sqrt[y/x]], Assumptions -> x > y > 0]
The answer is 0 indicating that they are indeed equivalent with the given assumptions.
We are left with the interesting question as to why we get two different, but equivalent, expressions when we merely swap the symbols.
I πand integration results can always differ by a constant. $\endgroup$