I believe, a lot of those questions comes from the lack of good TDD experience. I used TDD for almost 10 years already, so hopefully this answer will add a value to this question.
First things first. Here are some axioms (IMHO):
- unit tests can be well-engineered and poorly-engineered as any other code
- mocks simplify unit testing and efforts to support both a unit and unit tests
Now, having well-written unit tests allows you to have up-to-date view of how a unit can be used in different scenarios, including edge-cases. This is extremely important because having no unit tests does not provide you with such information. Imaging you was working on a feature, understood and implemented all quirks, edge-cases and fixed a dozens of bugs. But you haven't written any unit tests. What will happen when you leave a company? Do you expect new comers to grasp all of those details? They will suffer while debugging and understanding what you already understood. The project may struggle as well.
Next, what are well-written unit tests? Those are tests which facilitate refactoring. What does it means? It means that they atomic enough, so you don't need to spend more than 5-10 minutes to understand what each test does. It also means, that if business requirements are changed you don't need to rewrite everything. This is combines with how well-written, structured and architected the code base is as well. In well-designed application, fully written using TDD, making a change is a breeze. Sometimes new requirements does require removing a unit and all it's unit tests, but usually they small enough, so you don't feel a great loss.
My question is, how does the first of these approaches (ie sticking to the pyramid of tests and covering the code by lots of unit tests that use mocks) help grant confidence in correctness of the code and enable fearless refactoring?
Having that said, you covered all units with unit tests. And you MOCKED all dependencies. Why mocked? Without mocking you have too many dependencies and side-effects in the unit, you don't want your code be affected by a third-party code. Each unmocked dependency is potentially a hole in unit tests which may require more efforts from you on supporting them. An unmocked dependency may be considered as a tech debt. But don't be an idealist, sometimes it's OK to have unmocked dependencies. Such tech debt may be resolved in the future if something goes wrong with your logic or you'll see that the tests became harder to support.
Bear with me, the question is almost answered. Now the "pyramid of tests" comes. You have well-written unit tests. But how do you know that all those small pieces of logic work as you expect in the application? You write an integration tests, which do verify how units behave together (but not how - which is responsibility of unit test). It's another skill to write a well-written integration tests as well. And then you may add e2e tests.
Where is my error in my reasoning that unit tests grant little confidence and are hostile to refactoring, as opposed to integration tests?
It's not "that unit tests grant little confidence", it's poorly-written unit tests with poorly-engineered system grant little confidence. Integration tests involve many dependencies which are not mocked, and that is why it's difficult to write and support them. That's why you have much less integration tests, compared to unit tests. You have thousands of unit tests and dozens of integration tests. And you have some end to end tests. When it comes to refactoring. In a project with well-separated layers of a test pyramid, you much more often modify individual unit tests, and much less you modify integration tests (often you don't need to touch them at all). And you don't afraid to do so, because all you little edge-case scenarios are covered by tests, which means if you update the logic, you'll see which ones are failing and no longer satisfy the requirements. And then you decide whether it's a bug in your new logic or the tests must be updated.
Unit tests are like your best friends or team-mates which help you to do your job much more confidently by telling you "Hey, remember this small AC? You broke it, don't forget to implement it as well".