Finding the Position of the Minimum Neighbor for Each Element in a Large Matrix

I have a large matrix ($6020\times 8452$), and I want to create a new matrix where the relative position of the minimum neighbor is each element in the matrix. For example, if I had the matrix

$\begin{bmatrix} 1 & 2 & 3\\2 & 3 & 4\\3 & 4 & 5\end{bmatrix}$,

I want the program to return

$\begin{bmatrix} 9 & 4 & 4\\2 & 1 & 1\\2 & 1 & 1\end{bmatrix}$.

To explain, $9$ represents that the specific element is the minimum of all its neighbors; $4$ represents that the neighbor directly West of the specified element is the minimum of all neighbors; $2$ represents that the neighbor directly North of the specified element is the minimum of all neighbors; and $1$ represents that the neighbor directly Northwest of the specified element is the minimum of all neighbors. The numbering scheme is not an important aspect of the problem, I just used this scheme in order to make the question more understandable.

My problem is that the program I made takes too long to be a reasonable way to work with my matrix. Is there some way to speed up this program a significant amount?

Also, a few quick notes about the matrix. The matrix is about half zeros, so I am using a SparseArray. Moreover, the maximum value of the whole matrix is approximately $397$. The minimum value is approximately $237$.

matrix = ArrayPad[dataSet, 1, 500.]; Table[Position[{matrix[[i - 1]][[j - 1]], matrix[[i - 1]][[j]], matrix[[i - 1]][[j + 1]], matrix[[i]][[j - 1]], matrix[[i]][[j + 1]], matrix[[i + 1]][[j - 1]], matrix[[i + 1]][[j]], matrix[[i + 1]][[j + 1]], matrix[[i]][[j]]}, Min[{matrix[[i - 1]][[j - 1]], matrix[[i - 1]][[j]], matrix[[i - 1]][[j + 1]], matrix[[i]][[j - 1]], matrix[[i]][[j + 1]], matrix[[i + 1]][[j - 1]], matrix[[i + 1]][[j]], matrix[[i + 1]][[j + 1]], matrix[[i]][[j]]}]], {i, 2, Length[matrix] - 1}, {j, 2, Length[matrix[[1]]] - 1}] 

Thanks!