This may not be the cleanest or most efficient code, but it gets you what you are looking for.

Considering:

mat = {{0.2, 0.1, 0.01, 1, 1}, {0.1, -0.5, 0.4, 1, 2}, {0.4, -1.1, -0.1, 2, 1}, {0.7, 1., 1., 2, 2}, {0.3, 0.5, 0.6, 3, 1}, {0.9, 1.1, 0.6, 3, 2}}; mat // MatrixForm 

Now using Select you can link multiple criteria for each row. I am using Abs to streamline <-1 or >1.

criteriamet = Select[mat, Abs[#[[1]]] > 1 || Abs[#[[2]]] > 1 || Abs[#[[3]]] > 1 &]; criteriamet // MatrixForm 

To put the remaining rows into their own matrix, you simply have to Select those who are not members of the list you just made.

criterianotmet = Select[mat, MemberQ[criteriamet, #] == False &]; criterianotmet // MatrixForm 

There are certainly ways you can improve the code, but this should get you what you are looking for.

The basic framework for your question is relatively simple. First you need to select items from a list based on multiple criteria, and then you need to calculate which items from your list were not included. In this case, the matrix formatting is irrelevant.

Your original code is simply a list of lists and then displayed a matrix:

mat = {{0.2, 0.1, 0.01, 1, 1}, {0.1, -0.5, 0.4, 1, 2}, {0.4, -1.1, -0.1, 2, 1}, {0.7, 1., 1., 2, 2}, {0.3, 0.5, 0.6, 3, 1}, {0.9, 1.1, 0.6, 3, 2}}; mat // MatrixForm 

The key step is selecting the rows that meet your criteria. Using Select you can link multiple criteria for each row by using || to indicate "OR". (I am using Abs to streamline <-1 or >1).

criteriamet = Select[mat, Abs[#[[1]]] > 1 || Abs[#[[2]]] > 1 || Abs[#[[3]]] > 1 &]; criteriamet // MatrixForm 

To put the remaining rows into their own matrix, you simply have to Select those who are not members of the list you just made.

criterianotmet = Select[mat, MemberQ[criteriamet, #] == False &]; criterianotmet // MatrixForm 

There are certainly ways you can improve the code for clarity and efficiency, but this should get you what you are looking for.