IMPORTANT: Please look at the other answers. This is old and inaccurate answer.
You cannot chroot into different architecture. By chrooting, you are executing the binaries (from the chroot) on your architecture. Executing ARM binaries on x86 (and x86_64 in that matter) would lead to "Exec format error".
If you want to run binaries from different architecture you will need an Emulator. Qemu is a good candidate for this, but you will need to learn how to use it. This would involve creating RootFS and compiling a kernel for ARM. You will need a toolchain for compiling ARM binaries (and kernel) perhaps. One thing is for sure: Forget the chroot method, you cannot run binaries compiled for ARM on x86 (x86_64).
Edit: After the small talk with @UrichDangel, I realized, it should be possible to enter the chroot environment with qemu-user programs (qemu-arm in this case). Chroot should be executing qemu-arm compiled for your host architecture, then the qemu-arm can execute your /bin/sh (compiled for arm).