I'm studying VertexNormals (on Mathematica 10.0.2 for Mac) and have set up the following to represent three points and a normal vector:
p = {{1, 1, 0}, {1, 0, 1}, {0, 1, 1}}; n = {1, 1, 1}; and the following function to display a triangle whose vertices all have n as their normal direction:
g[p_] := Graphics3D[ { Yellow, Polygon[p, VertexNormals -> {n, n, n}], MapThread[{#1, Sphere[#2, 0.1]} &, {{Red, Green, Blue}, p}] }, Axes -> True, AxesLabel -> {x, y, z}, ViewPoint -> 100 {1, 1, 1} ]; The first vertex is marked by a red sphere, the second with a green one, and the third with a blue one.
g[p] produces

whereas g[Reverse@p] produces

Clearly, the normal direction actually depends on the orientation of the polygon. Is this an expected behaviour? I find this to be against intuition. If for example I specify that the normal direction for a particular vertex is along $\hat\imath + \hat\jmath + \hat{k}$, I'd rather expect that it always points away from the origin regardless of where other vertices are. This does not seem to be the case, and the documentation for VertexNormals doesn't say anything about this.
Could anyone please enlighten me as to why VertexNormals is designed to behave as such?
And given that such behaviour can't change, what's the computationally cheap way of choosing the right direction for vertex normals? The cheapest one that I can think of is, as described in the comment,
g[p_] := Graphics3D[ { Yellow, Polygon[p, VertexNormals -> If[ VectorAngle[Cross[p[[2]] - p[[1]], p[[3]] - p[[1]]], n] < Pi/2, 1, -1 ]{n, n, n}], MapThread[{#1, Sphere[#2, 0.1]} &, {{Red, Green, Blue}, p}] }, Axes -> True, AxesLabel -> {x, y, z}, ViewPoint -> 100 {1, 1, 1} ]; 



Polygon[p]isc = Cross[p[[2]] - p[[1]], p[[3]] - p[[1]]]. Suppose that a user specifies that a vertex normal for one the polygon's vertices isn = {n1, n2, n3}. It seems if the angle betweencandnis acute, Mathematica choosesnfor the direction; and if obtuse,-n. $\endgroup$Sign@Det@Prepend[Differences[p[[;; 3]]], n]yields1if the angle betweennand the orientation normal ofpis acute and-1if obtuse. You can use justpif the polygons are always triangles. I don't know if that's a cheap way or not. $\endgroup$