Here's a rough idea using image processing transformations to isolate the features of interest:
- Apply a flood fill from an ocean cell to make a mask of all ocean cells. Depending on how your rivers are set up, you might need an extra elevation or clearance criterion to keep the ocean mask from flowing inland. ;)
Apply a local smoothing to the edge of this mask, keeping the connectedness/topology the same, but smoothing out small noisy coastline features that can be distracting. This lets us focus on big bays over tiny inlets. You can use the width of your filter kernal / number of iterations to finely control the scale of features you preserve.
Here I applied a median filter a few times. Cellular automata are another popular way to erode smooth shapes from a noisy input.
- Turn the mask into a distance field, where each cell stores its distance from the smoothed coastline.
Now we see some promising feature highlights. In a signed distance field, both bays and straits show up as sharp ridges, with distance falling off to the sides. We can use an edge detection filter to pop out these ridges:
You can then distinguish between bays and straits by following the ridge to determine its connectivity. A bay is a ridge that runs toward the coast, getting shallower and shallower (in distance from land) until it ends in a point. A strait is a ridge that connects a high-distance region to another high-distance region, going through a lower-distance region along the way.
Or, another way is to assign each island an ID (connected component search), then when you're making your distance field, propagate "closest island ID" alongside the distance frontier. A bay or inlet is then a ridge in water adjacent to the same landmass on both sides, while a channel is a ridge that separates water adjacent to two different landmasses.
You can set minimum & maximum distance-to-shore or length-of-ridge constraints to control which features to label, if you need to exclude excessively narrow/wide straits for instance.




