I hit the same issue as you. It seems a fairly rare use case, as this is pretty much the only result that comes up for it on Google.
I did eventually find the issue though, and it is to do with mounting /proc into the chroot (from within the Docker container)
In your example, prior to entering/using the rootfs you should mount Docker's /proc into the chroot:
mount -t proc /proc <armbian-rootfs-dir>/proc You will then get native speeds for cryptography within your chroot