I would strongly recommend starting with nursesncurses.
Unlike more complex windowinggraphical systems, it is based purely on text, so there is no need to get bogged down in the details of screen drivers and graphics libraries. However the basic principles of putting windows on a screen, moving focus between windows, and so on, still hold true. And you can still do some drawing, at the level of single character blocks and ASCII art.
Of course you're still building this on top of a library, but it's a library which you can easily understand. And more than that, it's a library where the source code is freely available, fairly well documented, and not too impenetrable if you want to read it. You can even modify it yourself if you want to. Or you could look at all the library functions in there to find what the API needs to be, and write it yourself from scratch based on that design.