Prove that $a^2+b^2+c^2+5abc\geq8$

Case $a+b+c>4:$ Multiplying both sides by $a+b+c,$ the inequality becomes $$(a+b+c)(a^2+b^2+c^2)+5abc(a+b+c) - 8(a+b+c) \geqslant 0.$$ Since $a+b+c>4,$ we can add $-20abc$ to $5abc(a+b+c)$ to form as $$[(a+b+c)(a^2+b^2+c^2)+20abc - 8(a+b+c)] + 5abc(a+b+c-4) \geqslant 0.$$ We need to prove $$(a+b+c)(a^2+b^2+c^2) + 20abc \geqslant 8(a+b+c).$$ Let $t = a+b+c > 4,$ then $\frac{t^3}{2} \geqslant 8t.$ It remain to prove that $$(a+b+c)(a^2+b^2+c^2) + 20abc \geqslant \frac{(a+b+c)^3}{2}.$$ Expand and simplify to $$a^3+b^3+c^3+34abc \geqslant ab(a+b)+bc(b+c)+ca(c+a).$$ Which is true by Schur’s inequality.

I think your idea is good because $\prod\limits_{\rm cyc}(a-b)^2\ge0$ is strong enough, but the following approach can avoid radicals.

Let $p=a+b+c$, $q=ab+bc+ca$, $r=abc$, then by AM-GM inequality and the condition, we have $p^2\ge3q\ge(p-2)^2+8$ (this yields $p\ge3$). For the sake of contradiction, assume that $$\sum_{\rm cyc}a^2+5abc=p^2-2q+5r<8,$$ so $r<\frac{8+2q-p^2}5$ (this yields $q\ge\frac{p^2-8}2$).

Consider $$-\prod_{\rm cyc}(a-b)^2=27r^2+\left(4p^3-18pq\right)r+4q^3-p^2q^2=:f(r).$$ The symmetry axis of this quadratic function is $r=\frac{9pq-2p^3}{27}$. Notice that $$\frac{9pq-2p^3}{27}-\frac{8+2q-p^2}5=\frac1{135}\left[(45p-54)q-10p^3-216+27p^2\right]\\ \ge\frac1{135}\left[(15p-18)\left(p^2-4p+12\right)-10p^3-216+27p^2\right]\\ =\frac{(p-3)\left(5p^2-36p+144\right)}{135}\ge0, $$ so $f(r)$ is a decreasing function when $r<\frac{8+2q-p^2}5$. Hence, $$f(r)>f\left(\frac{8+2q-p^2}5\right)\\=\frac1{25}\left(\begin{aligned}1728-432 p^2+160 p^3+27 p^4-20 p^5\\+864 q-720 p q-108 p^2 q+130 p^3 q\\+108 q^2-180 p q^2-25 p^2 q^2+100 q^3\end{aligned}\right)=:g(q).$$ We have $$g'(q)=\frac2{25}\left(432-360 p-54 p^2+65 p^3+108 q-180 p q-25 p^2 q+150 q^2\right),$$ of which the symmetric axis of $q$ is $q=\frac{25p^2+180p-108}{300}$. Since $\frac{25p^2+180p-108}{300}\le\frac{(p-2)^2+8}3$, then $$g'(q)\ge g'\left(\tfrac{(p-2)^2+8}3\right)=\frac{2(p-3)}{75}\left(1736 p-210 p^2+25 p^3-3264\right)\ge0,$$ which means that $g(q)$ is an increasing function. Therefore, $$g(q)\ge\min\left\{g\left(\frac{(p-2)^2+8}3\right),g\left(\frac{p^2-8}2\right)\right\}\\ =\min\left\{\begin{aligned}&\frac{p-4}{675}(p-3)^2 \left(1696 p-260 p^2+25 p^3-9984\right),\\&\frac{p-4}4(4+p)\left(p^2-8\right)^2\end{aligned}\right\}\ge0,$$ which is because the former is non-negative when $p\in[3,4]$ and the latter is non-negative when $p\ge4$.

All of the above indicates that $\prod\limits_{\rm cyc}(a-b)^2<0$, contradiction! We are done.

The pqr method for the case $a + b + c > 4$.

Let $p := a + b + c, q := ab + bc + ca, r := abc$. We need to prove that $p^2 - 2q + 5r \ge 8$.

If $p^2 \ge 4q$, we have $$p^2 - 2q + 5r - 8 \ge p^2 - 2\cdot \frac{p^2}{4} - 8 = \frac12 p^2 - 8 \ge 0.$$

If $p^2 < 4q$, using degree three Schur inequality $r \ge \frac{4pq-p^3}{9}$, it suffices to prove that $$p^2 - 2q + 5\cdot \frac{4pq-p^3}{9} \ge 8, $$ or $$(20p/9 - 2)q - 5p^3/9 + p^2 - 8 \ge 0.$$

Since $20p/9 - 2 \ge 0$, using $q \ge p^2/4$, it suffices to prove that $$(20p/9 - 2)\cdot \frac{p^2}{4} - 5p^3/9 + p^2 - 8 \ge 0,$$ or $$\frac12p^2 - 8 \ge 0$$ which is true.

My first solution.

It suffices to prove that, for all $a, b, c \ge 0$, \begin{align*} &a^2 + b^2 + c^2 + 5abc - 8\\ &\qquad - \frac14(a + b + c + 4)\left(3ab + 3bc + 3ca - (a+b+c-2)^2 - 8\right)\ge 0. \tag{1} \end{align*}

We use the pqr method. Let $p := a + b + c, q := ab + bc + ca, r := abc$. (1) is equivalently written as $$(-3p-20)q+p^3+4p^2-4p+20r+16 \ge 0. \tag{2}$$

If $p^2 \ge 4q$, we have \begin{align*} &(-3p-20)q+p^3+4p^2-4p+20r+16\\ \ge{}& (-3p-20)\cdot \frac{p^2}{4} + p^3+4p^2-4p+20\cdot 0+16\\ ={}& \frac14(p+4)(p-4)^2\\ \ge{}& 0. \end{align*}

If $p^2 < 4q$, to prove (2), using degree three Schur inequality $r \ge \frac{4pq-p^3}{9}$, it suffices to prove that $$(-3p-20)q+p^3+4p^2-4p+20\cdot \frac{4pq-p^3}{9}+16 \ge 0$$ or $$-11p^3/9 + 4p^2 - 4p + 16 - (20 - 53p/9)q \ge 0. \tag{3}$$

(i) If $20-53p/9 \ge 0$, using $q \le p^2/3$, it suffices to prove that $$-11p^3/9 + 4p^2 - 4p + 16 - (20 - 53p/9)\cdot p^2/3 \ge 0,$$ or $$\frac{4}{27}(5p+12)(p-3)^2 \ge 0$$ which is true.

(ii) If $20 - 53p/9 < 0$, using $q > p^2/4$, it suffices to prove that $$-11p^3/9 + 4p^2 - 4p + 16 - (20 - 53p/9)\cdot p^2/4 \ge 0,$$ or $$\frac14(p+4)(p-4)^2 \ge 0$$ which is true.

We are done.

Hint:

We could show that $a^2+b^2+c^2+ 5 a b c\le 8$ and $a$, $b$, $c\ge 0$ implies $(a+b+c-2)^2+8\ge 3(ab+ac+bc)$, and find the equality cases.

Some facts easy to check:

1. The region given by $a^2+b^2+c^2+ 5 a b c\le 8$ and $a$, $b$, $c\ge 0$ is compact.

2. The region $(a+b+c-2)^2+8\le 3(ab+ac+bc)$ is convex.

From the above, we conclude that is enough to prove the inequality if $a^2+b^2+c^2+ 5 a b c= 8$. We now have an extremum problem with constraints. There are two cases:

1. one of the $a$, $b$, $c$ is $0$, say $c=0$.

2. there exists $\lambda$ such that the Lagrange system below is satisfied

$$2(a+b+c-2) - 3 (b+c) - \lambda(2 a -5 b c) = 0\\ 2(a+b+c-2) - 3(a+c) - \lambda( 2 b - 5 a c) = 0\\ 2(a+b+c-2) - 3(a+b) - \lambda( 2 c - 5 a c) = 0$$

From the above we conclude that $a$, $b$, $c$, are all solutions of an equation of the form $\frac{\alpha}{x} + \beta x + c = 0$. Now such an equation has at most $2$ positive solutions, hence two of the $a$, $b$, $c$ are equal. Therefore, we reduce to the case $b=c$.

so the cases are really

1'. one of the $a$, $b$, $c$ is $0$

2'. the $a$, $b$, $c$ take only two values

( this works in many other cases of symmetric extreme problems).

We'll skip for now the details for the problems such reduced, just note that the equality cases are $(1,1,1)$, and permutations of $(0,2,2)$.