Factor + math.factorials, 4141 32 bytes
Saved 9 bytes thanks to @Bubbler!
[| x | 0 [ 1 + dup factorial[1,b] x[ <n! = ] loopwith find ] Factor, 5050 34 bytes
Saved 16 bytes thanks to @Bubbler!
[| x | 0 1 [ swap 1 +over dup[1,b] rot[ * dup x2dup <= ] loop .find ] There's probably a better idiom for this, but I can't find it right now.