First, separate messages from their handlers.
Instead of having MessageTypeA handle its own parsing to and from binary; introduce an intermediary MessageHandlerTypeA which does this. This way you can reduce the messages themselves to Plain Ol' Data types or - at the very listleast - simple data containers, and you can keep the actual handlers instanced throughout instead of creating them new each time.
Then in your message receiving code you can register for each message type which MessageHandler to pass the binary data to. This is extensible from outside the library since you can then expose the registration function to allow new messages to be arbitrarily interpreted.
Second, use callbacks to do further message handling.
Instead of the receive and then process approach you are using, have the MessageHandler decode the message into the appropriate message type and then call any code that needs to know about that particular message. Your calling code, then, rather than interpreting the message simply registers a callback with the handler and then receives a callback with the interpreted message whenever that message is received.