In that single equation for \$V_{OUT}\$, there are four unknowns: \$R_1\$, \$R_2\$, \$V_{OUT}\$ and \$I_1\$. This means that you need four independent equations, to solve for any of them, which you don't have, yet.
Since your intention is to solve for \$R_1\$ and \$R_2\$, you must obtain two independent equations in \$R_1\$ and \$R_2\$ only, which can be obtained by substituting known values for \$V_{OUT}\$ and \$I_1\$.
One pair of values is already known. When \$I_1=0A\$, then \$V_{OUT}=100mV\$. By plugging those into the equation, you will have one of the two necessary equations in \$R_1\$ and \$R_2\$.
However, you will also need another value for \$I_1\$, and the corresponding \$V_{OUT}\$, to obtain the second equation, which you have not specified. Only then will you have the two equations, with two unknowns, that can be solved.
The schematic you provided does actually give you two pairs of values. The first pair would be \$I_1 = 0.4mA\$ and \$V_{OUT} = 0.4V\$. The second pair is \$I_1 = 20mA\$ and \$V_{OUT} = 1.6V\$.
Using your equation, this yields:
$$ \begin{aligned} 0.4 &= \frac{2.5(R_2+10k)+4m \times R_1 \times R_2 }{R_1+R_2+10k\Omega} \\ \\ 1.6 &= \frac{2.5(R_2+10k)+20m \times R_1 \times R_2 }{R_1+R_2+10k\Omega} \\ \\ \end{aligned} $$
These I rearranged by multiplying both sides by \$R_1+R_2+10k\Omega \$, and expanding:
$$ \begin{aligned} 0.4R_1+0.4R_2+4000 &= 4\times 10^{-3}R_1R_2 + 2.5R_2 + 25000 \\ \\ 1.6R_1+1.6R_2+16000 &= 20\times 10^{-3}R_1R_2 + 2.5R_2 + 25000 \\ \\ \end{aligned} $$
To eliminate the \$R_1R_2\$ term I multiplied all the terms in the first by 5, and subtracted corresponding terms from the second, to get:
$$ \begin{aligned} 0.4R_1+0.4R_2+4000 &= 10R_2 + 100000 \\ \\ \end{aligned} $$
This can be manipulated to get \$R_1\$ purely in terms of \$R_2\$ (or vice versa), which you can then substitute back into a previous equation, to solve. I did indeed find a quadratic equation, which gave me two solutions:
$$ \begin{aligned} R_2 &= 78.125 \text{ or } -10000 \\ \\ R_1 &= 241875 \text{ or } 0 \\ \\ \end{aligned} $$
Whatever technique you use to solve them can involve a quadratic expression in \$R_1\$ and \$R_2\$, with two possible pairs of values for \$R_1\$ and \$R_2\$, and one of those pairs may contain negative (or even complex) values.
It's only quadratic in \$R_1\$ and \$R_2\$, though, not \$I_1\$ or \$V_{OUT}\$. After you've found \$R_1\$ and \$R_2\$, these are constant values, which when plugged into the original equation yield a linear relationship between \$I_1\$ and \$V_{OUT}\$.
Both pairs of results you obtain for \$R_1\$ and \$R_2\$ are algebraically legitimate, although only the positive, real solution is feasible in reality, using real components.
