Let's define "access the port" first.
A serial/USB port will have a special IC on the mother board. It will be connected by electrical traces to a peripheral controller or other ICs, all eventually wired to the CPU.
So, when we say "access the port" to mean "control the electrical signals on those traces wired to that USB controller", then yes a kernel mode driver can make that happen.
The problem is when we define "access the port" to mean "make the port do something useful". A USB controller is a very complex chip, the USB protocol is a huge pain to implement. You're not going to be able to guess the right way to talk to it, you're not going to guess how the chip interprets the electrical signals applied to its pins. Sometimes the makers of the IC will keep those details secret, and even if they publish them, it can be a lot of work.
In the case of USB, there exists a company, FTDI, where its entire reason for existence is to allow people to make USB connected hardware and simultaneously avoid the hassle of writing drivers for it.
So the answer to your question is probably no, because while your written question is in the context of the first definition, you're thinking and your goals are of the second.
