Territories are Account based, but Forecasts are Opportunity based. With Enterprise Territory Management (TM2), there can be many different structures to Account Territories, not all of which are strictly based on a single hierarchy. Account assignment rules can be geographic-based, product-based, customer-size based (in terms of number employees, sales, etc), and they can also be manually assigned without reason or rhyme.
Similarly, Opportunities have their own assignment rules, including manual overrides. In that sense, you could say that Opportunity (think Forecasts here, since Opportunity is what Forecasts are based on), has it's own mishmash of hierarchies.
Hopefully you're familiar with the term "Account Teams" and also with "Opportunity Teams". One manages relationships with Accounts and the other focuses on new deals or Opportunities with both existing and new Accounts.
Just as you have an Account Hierarchy in Territories, because of having a different set of rules for assigning Opportunities to Sales People independent of how Accounts are assigned, you'd get a different set of forecasts based on Opportunity Territory assignment rules than you would based on Account Territory assignment rules.
I hope that adds at least some clarity to the differentiation. How it precisely applies greatly depends on the organization and their specific rules for assigning Accounts and Opportunities.