I don’t know if there’s another term of art for this, but it’s a kind of topological sorting by a partial ordering. This comes up in pattern matching, generic function instantiation, and multimethod dispatch, particularly when you want to find the unique most specific matching pattern for a given input. A special case in typechecking is subsumption, where you want to check that a definition’s inferred type is at least as general as its declared type signature.
You have a partially ordered set (poset) of patterns, where $P < Q$ if $P$ is less general than $Q$, meaning that $Q$ matches at least everything $P$ does. In other words $P$ is more specific than $Q$. Incomparable elements are patterns like (regex) a. and .b where neither one matches everything that the other does. All atomic patterns like a and b are incomparable in this way. They’re also “atomic elements” of the poset, meaning minimal among the nonzero elements.
You can define this ordering in various ways depending on the pattern language. For a regular language, you can consider $P < Q$ when their difference (intersection with complement) is empty: $P \setminus Q = P \cap \neg Q$ and $(P \setminus Q = \emptyset) \iff (P \subseteq Q)$. Computing this is decidable in polynomial time, for example using the Brzozowski derivative.
However, it’s easy for large inputs or extensions to the pattern language to make that method intractable or undecidable, so it’s more typical to define an ordering recursively. You can handle some cases precisely, like any pattern is less general than a wildcard $a < \top$, and ordering distributes over sequences of patterns, $ab < cd$ if $a < b$ and $c < d$. Other cases like choice $a+b \stackrel{?}{<} c+d$ you might handle heuristically.
Standard toposort algorithms apply, although I’ll note that if you have a term, you don’t need to sort the whole set of patterns to find the best matching pattern for it. Instead you can just iterate over the patterns while keeping a set of the most specific incomparable matching candidates seen so far. Whenever you find a match more specific than all of them, it becomes the new sole candidate. In the end you’ll have an ambiguity (>1), a unique match (=1), or no match (<1).
You may also be interested in the next step, Compiling pattern matching to good decision trees(PDF) (Maranget 2008).
