1)Find barycentric coordinates
for(int i = 0; i < verticesInPolygon; i++) { bary.xyz += vertices[i].xyz; } bary.xyz /= verticesInPolygon;
2)Create new vertices using this formula
newVertex.coord = (vertices[desiredVertexInPolygon].coord - bary.coord)*offset+bary.coord;
Where coord is a desired coordinate(x, y or z) and offset is a scalar determining how far from the barycenter and how close to the original vertex a new vertex will be, i. e. 0.5 would make it appear right between, making new polygon being a half size of the original.

UPDATE: So, you need a constant offset to make outline, right? Here is a way to do it.
1)Find barycentric coordinates
We already know how to do this.
2)Find a unit direction vector from barycenter to the original vertex
dir.xyz = normalize(vertices[desiredVertexInPolygon].xyz - bary.xyz);
3)Create new vertices with a following formula
newVertex.xyz = vertices[desiredVertexInPolygon].xyz + dir.xyz * offset;
Where offset is now a real distance rather than scalar.

Note that in your case you should use a negative offset value, because you want outline to be inside the polygon.