When you say you want to calculate the "distance for A*", you mean the heuristic, right? The heuristic is an estimate of the distance. And you want to avoid overestimating it. See Admissible heuristic.
Unless we are dealing with teleportation, or some not euclidean geometry, there is no way that units will reach the destination via a shorter path than a straight line. Thus, the distance on straight line never overestimates the distance. Therefore the distance on straight line is an admissible heuristic. Which means A* should work correctly using it.
I mean, something like this:
distance_estimate = (dst - src).length()
However, if I calculate the distance between centers of navmeshes when I use A*, the path would be calculated as red in the picture.
This could happen if you overestimated the distance. The distance from the centers of the navmeshes can be longer than the actual distance, so that can happen. Although you still got a valid path.
By the way, I imagine you are doing something like this:
var src_area = get_area(src) var dst_area = get_area(dst) distance_estimate = (dst_area.center - src_area.center).length()
So, how do I account for entry and exit point in navmeshes when calculating the distance for A*?
You can pick the exit that is closer to the other point. Closer as measured in a straight line. That might not be the correct exit, but it would avoid overestimating the distance (assuming you have convex areas, which is how these systems work). Which makes it an admissible heuristic.
I mean, something like this:
var src_area = get_area(src) var src_exit = get_min(src_area.exits, exit => (exit - dst).length()) var dst_area = get_area(dst) var dst_exit = get_min(dst_area.exits, exit => (exit - src).length()) distance_estimate += (dst_exit - src_exit).length()
However, it is not necessarily a better heuristic. We want to make sure we don't overestimate, but we want to underestimate as little as possible. We can make it better if we add the distance from the point to the exit we picked.
So, you would do something like this:
var src_area = get_area(src) var src_exit = get_min(src_area.exits, exit => (exit - dst).length()) var dst_area = get_area(dst) var dst_exit = get_min(dst_area.exits, exit => (exit - src).length()) distance_estimate = (src_exit - src).length() + (dst_exit - dst).length() + (dst_exit - src_exit).length()
