I'm not going to address the design problem - just the question of whether to do things "correctly" in a non-public API.
it's just for me, so it's kind of like I'm protecting myself from my own code from myself
That's exactly the point. Maybe there's coders out there who remember the nuances of every class and method they ever wrote and never mistakenly call into them with the wrong contract. I'm not one of them. I often forget how code I wrote is supposed to work within hours of writing it. After you think you've gotten it right once, your mind will tend to switch gears to the problem you're working on now.
You have tools to combat that. These tools include (in no particular order) conventions, unit tests and other automated tests, precondition checking, and documentation. I myself have found unit tests to be invaluable because they both force you to think about how your contract will be used and provide documentation later on how the interface was designed.