It depends on the type of application and the type of market you are in.

There are trade-offs and benefits to going this way. It is not a clear-cut answer that one way is better than the other.

I'll talk from personal experience. I was the one who decided to take the codebase that I work on in this direction back in 2007. That codebase is somewhere in the order of a million lines of code now, half of which is server code hidden behind a massive amount of web service API's, the other half is a flotilla of clients, desktop native, desktop web, mobile, back-end integrations, etc... This decision was not without its downsides, but with 20/20 hindsight I can say I would do it again. Let me indicate some of the trade-offs involved.

Benefits

• Flexibility. Whether it's a request to build a mobile app to augment the desktop experience, or a request to integrate with SAP's back-end, it all becomes easier when you already have an API to call. When you get enough of these requests, you will organically evolve towards an API, and the only question is whether that has a standard web service in front of it, or whether it is an internal API where the web services are tailor-made.

• Scalability (of the team). In our case we have many different groups of developers all building on top of this API. We even have dedicated API teams, who talk to the different groups, summarize the needs, and construct an all-purpose API out of it. It's gotten to the point where we're not even told anymore that people are building stuff on top of the API, and not everyone who does that works for our company.

• Security. Having a clear-cut division between the unsafe and safe parts of your codebase is helpful in reasoning out security. Muddling the UI and back-end code together tends to confuse matters.

Trade-offs

• Flexibility. You have to do the work to "properly" build something into the API. It's not possible to quickly run a DB query from inside the UI code to solve a specific problem. Also, API's which are actually reusable must take so many use-cases into account that the quick solution is usually the wrong solution. The API becomes less flexible to evolve, especially since there is so much client-code already out there (we're transitioning to a versioned API for that reason).

• Initial development speed. It is slower to develop API-first, without a shred of a doubt. You only win it back when you have enough clients built on top of the API. But then you find that you need 3 different client implementations before your API has evolved to be generic enough. We found that most of our initial API designs were wrong and have had to strongly revise our guidelines for how to build web services.

Red herrings

You mentioned a bunch of these. They don't actually matter in practice.

• Abstraction. Your API becomes abstract enough to cover all use cases that your product needs to serve, and no more than that. Even without web services you'll either have an internal API which does this, or have lots of duplicate code. I prefer abstraction over duplication.

• Abandoning the server-side MVC stack. These days almost every system will after a while need a mobile app. When you then build web services to cater to that mobile app, you will have to figure out how to do authentication and authorization in an API context anyway. It is actually less work when you only have one way of doing that, the way you do it in your web services.

• Bulk operations. Typically solved by creating a bulk API which launches a back-end job and returned a job id for status querying. It's not that big of a deal.

• Debugging. I found that on the whole it became slightly easier to troubleshoot the system. You can still set breakpoints in both front-end and back-end code, so in practice it's not that harder to step through, and you gain the ability to build automated api tests and to instrument the api for monitoring production systems.

• Lots of independent operations. That's a matter of how you design things. If you insist on having a pure CRUD API, then yes, you will suffer from this issue. But having some CQRS API's to augment that is typically a good idea, and if you have made sure that you have an internal API for which the services are a front-end, then you can easily reuse that internal API to construct services for those specific scenario's.

In Summary

In a system that is used in enough different contexts, an API will naturally evolve as it is the easiest way to cater to all the needs. But there is definitely a case of YAGNI going on. There are trade-offs and it doesn't make sense until it makes sense. The key point is to not be dogmatic and to keep an open mind towards different approaches in architecture to meet the evolving needs of the product.