How to transition users from a mobile-optimized to non-optimized area of a site with minimum confusion?

Depending on the framework you're using, it is possible to transfer everything at once; we actually just did this a few weeks ago and outside of a few pages that were already problematic, it was fine.

That said, there's almost no way you can do it without causing confusion. My recommendation would be to automatically have a one-time web pop-up that tells new users on the mobile site that you're undergoing construction and that yes, the site has some pages that aren't mobile friendly just yet. That way users know what to expect. It's the false expectation that's the worst. People are still used to non-mobile friendly websites, and smartphones can all handle that just fine.

It may be more prudent for you to update your workflow to upgrade the site based around most valuable/most visited first, and follow those flows all the way through. I'd warn that you really have to complete the change too; more often than not people/companies stop making their sites mobile friendly purely because they covered the major areas.

I'd say of every UX aspect is the consistency in design.

Second, is to sort out the basics. You don't have to migrate everything to your mobile app. I don't know what kind of data you're dealing with but focus on core data. Content that really matter to your audience.

Test, iterate, launch, and test again.