qalc/Qalculate, 13 bytes

(ans−1)!²%ans 

The great thing about "programming" in a calculator is that you can basically just enter formulas directly.

Example usage in Bash:

echo "5" | qalc -t -f - "(ans−1)! ²%ans" 

Note that I added a space after the exclamation mark only because Bash otherwise tries to interpret "!²" as an "event". It's not actually required by qalc, which can be seen by for example creating this file:

5 (ans−1)!²%ans 

…and then executing it with qalc -t -f test.txt.

Other consoles or other ways of calling qalc than Bash might not have this problem.

The calculation can also be entered in "interactive mode" by simply running qalc -t, entering the number and then entering the expression on the next input. This also works in the GUI program "Qalculate".

Meta explanation:

The output is 0 for non-prime numbers and a positive integer for prime numbers. I let this run in a loop for all numbers up to 10000 and it was always 1 for prime numbers, but I haven't checked whether that is guaranteed yet.

This solution is based on this Labyrinth answer. I essentially just found something (qalc) that has all the parts of the formula as builtins instead of having to program all the parts myself on a lower level.

The -t argument hides the interpreted calculation, so that the output of the example command looks like this:

5 1 

…instead of like this:

5 = 5 rem(factorial(ans - 1)^2, ans) = 1 

The tricky part was reading the number in. It's pretty easy to insert the number into the formula two times, but that would mean modifying the "program", which is surely not allowed. There seems to also be no way to execute multiple expressions non-interactively with qalc, except by reading in a file. In this case, I used - as a "file", which is StdIn, and used that only to store the input in ans, which is implicitly done by qalc. You could also say that ans is the input of the "program", in which case the program is simply the 13 bytes seen at the top. It could also be stored in any other variable, in which case the solution could be shortened by 4 bytes.

Algorithm explanation:

The Wikipedia article linked in the Labyrinth answer looks pretty confusing, but the principle is easy:

(6-1)!²%6 = 1·1·2·2·3·3·4·4·5·5%6 = 1·1· 6 · 6 ·4·4·5·5%6 = 6 · something %6 = 0 

This works because every non-prime number is the product of at least two factors that are smaller than that number. Prime numbers aren't, which leads to something like this:

(5-1)!²%5 = 1·1·2·2·3·3·4·4%5 = 1·1·2·2·3·3·4·4%5 (no multiplication of those factors results in anything divisible by 5) = 1 

The squaring is necessary in case there is only one prime factor and that occurs twice:

(9-1)!²%9 = 1·1·2·2·3·3·4·4·5·5·6·6·7·7·8·8%9 = 1·1·2·2· 9 ·4·4·5·5·6·6·7·7·8·8%9 = 9 · something %9 = 0 

I originally wrote this answer for TI-Basic:

fPart((Ans-1)!²/Ans 

That's just 10 bytes, but it fails on the input 13 already, due to floating point imprecisions in those large numbers (12!²=229442532802560000). So the only difference to an even/odd checker within the working range is 9 and it does not fulfil the criterion to work up to 255. For inputs above 42 it even crashed from an overflow (42!²>10¹⁰⁰).