Let's make the change of variables $x=e^{-\xi}$, $t=e^{-\tau}$ in the integral equation: $$ \int_t^{1}f(x)f\!\left(\frac{t}{x}\right)dx=\int_0^{\tau}f(e^{-\xi})f(e^{-(\tau-\xi)})e^{-\xi}\,d\xi=e^{-\tau/2}. \tag{1} $$ Defining $g(z):=f(e^{-z})$, we can rewrite $(1)$ as $$ \int_0^{\tau}g(\tau-\xi)g(\xi)e^{-\xi}\,d\xi=e^{-\tau/2}. \tag{2} $$ Taking the Laplace transform of both sides of $(2)$, we get $$ G(s)G(s+1)=\frac{1}{s+\frac{1}{2}}, \tag{3} $$ where $G(s):=\mathcal{L}\{g(\xi)\}(s)$.
The positive solution to the functional equation $(3)$ is (see the Appendix) $$ G(s)=\frac{1}{\sqrt{2}}\frac{\Gamma\!\left(\frac{s}{2}+\frac{1}{4}\right)}{\Gamma\!\left(\frac{s}{2}+\frac{3}{4}\right)}. \tag{4} $$
Finally, taking the inverse Laplace transform of $G(s)$, we obtain$^{(*)}$ $$ g(\xi)=\sqrt{\frac{2e^{-\xi}}{\pi\left(1-e^{-2\xi}\right)}}, \tag{5} $$ hence $$ f(x)=\sqrt{\frac{2x}{\pi\left(1-x^2\right)}}. \tag{6} $$
Appendix: Solution of Eq. $(3)$
To solve Eq. $(3)$, let's rewrite $G(s)$ as $$ G(s)=\frac{G(s)G(s+1)\cdots G(s+2n-1)}{G(s+1)G(s+2)\cdots G(s+2n)}\,G(s+2n)\qquad(n>1). \tag{A.1} $$ Using Eq. $(3)$, we can rewrite $(\text{A.}1)$ as \begin{align} G(s)&=\frac{\left(s+\frac{3}{2}\right)\left(s+\frac{7}{2}\right)\cdots\left(s+2n-\frac{1}{2}\right)} {\left(s+\frac{1}{2}\right)\left(s+\frac{5}{2}\right)\cdots\left(s+2n-\frac{3}{2}\right)}\,G(s+2n) \\ &=\frac{\left(\frac{s}{2}+\frac{3}{4}\right)\left(\frac{s}{2}+\frac{7}{4}\right)\cdots\left(\frac{s}{2}+n-\frac{1}{4}\right)} {\left(\frac{s}{2}+\frac{1}{4}\right)\left(\frac{s}{2}+\frac{5}{4}\right)\cdots\left(\frac{s}{2}+n-\frac{3}{4}\right)}\,G(s+2n). \tag{A.2} \end{align} Now, using the identity $x(x+1)\cdots(x+n-1)=\frac{\Gamma(x+n)}{\Gamma(x)}$, we can rewrite $(\text{A}.2)$ as $$ G(s)=\frac{\Gamma\!\left(\frac{s}{2}+\frac{1}{4}\right) \Gamma\!\left(\frac{s}{2}+n+\frac{3}{4}\right)} {\Gamma\!\left(\frac{s}{2}+\frac{3}{4}\right) \Gamma\!\left(\frac{s}{2}+n+\frac{1}{4}\right)}\,G(s+2n). \tag{A.3} $$ Since the RHS of $(\text{A}.3)$ does not depend on $n$, we can simplify it using the fact that Eq. $(3)$ implies $G(s+2n)\sim\frac{1}{\sqrt{s+2n}}$ as $n\to\infty$. Taking this limit with the help of WolframAlpha, we finally obtain $$ G(s)=\lim_{n\to\infty}\frac{\Gamma\!\left(\frac{s}{2}+\frac{1}{4}\right) \Gamma\!\left(\frac{s}{2}+n+\frac{3}{4}\right)} {\Gamma\!\left(\frac{s}{2}+\frac{3}{4}\right) \Gamma\!\left(\frac{s}{2}+n+\frac{1}{4}\right)}\,\frac{1}{\sqrt{n+2s}} =\frac{1}{\sqrt{2}}\frac{\Gamma\!\left(\frac{s}{2}+\frac{1}{4}\right)} {\Gamma\!\left(\frac{s}{2}+\frac{3}{4}\right)}. \tag{A.4} $$ Direct substitution shows that $(\text{A}.4)$ indeed satisfies Eq. $(3)$: $$ G(s)G(s+1)=\frac{1}{2}\frac{\Gamma\!\left(\frac{s}{2}+\frac{1}{4}\right)\Gamma\!\left(\frac{s+1}{2}+\frac{1}{4}\right)} {\Gamma\!\left(\frac{s}{2}+\frac{3}{4}\right)\Gamma\!\left(\frac{s+1}{2}+\frac{3}{4}\right)} =\frac{\Gamma\!\left(\frac{s}{2}+\frac{1}{4}\right)} {2\,\Gamma\!\left(\frac{s}{2}+\frac{5}{4}\right)} =\frac{1}{s+\frac{1}{2}}. \tag{A.5} $$
$^{(*)}$ A.P. Prudnikov, Yu.A. Brychkov, O.I. Marichev, Integrals and Series, Volume 5: Inverse Laplace Transforms, formula $3.1.1.1$.
