It's not that hard to build a somewhat working RPG game mechanic, especially when there are so many inspirations around like pen and paper RPGs with Dungeons and Dragons being the prime example. But to balance your system, that's a different story. There are several facets of balance even for a simple RPG:

1. Are there overpowered or useless character development strategies? Like choosing level up stats|abilities in a way which guarantees more "gain" than all other ways.
2. Are there overpowered or useless character equipment|abilities|whatever else interchangeable on a character? Leveled comparison of course, so it's cracked wooden club versus rusted iron dagger and Gae Bolg versus Excalibur. Note the cost/benefit ratio.
3. Overall, is there one approach in a subsystem which is always better or worse than the others? Like one battle tactics which can ~always perform better than all another tactics.
4. Is the difficulty curve and challenge level appropriate?

I created a completely fluid system for a TRPG once where one could have any crazy combinations of abilities, stats and overall behaviour, so it was impossible to balance it manually very well. But we have an edge against pen-and-papers — ability to run lots of simulations on our fancy machines and not in our minds, so let's tackle those problems one by one. I won't give useless implementation specifics, just a general approach I've used:

1. Perturbate all gains for a level and assign each perturbation to some of your RPG role. Like if character gained +5 STR and Shield Bash ability, it becomes a fighter type. That'll be easier if your RPG have rigid classes. Then take snapshots of resulting character statistics and abilities per level and compare them one to another within those archetypes. That way you can see if one way of developing "fighter" is better than the other. Even on that stage you may notice that, for example, mages gain some unintended edge over fighters.

2. Build a combat simulator. Take one of your snapshots from a previous step and equip it with an item. Clone that shapshot and equip it with another item with the same "level range". Now run them through a combat simulator against enemy of that item's "level range" many times and check results. You may want to take abilities into account, then don't use fighter snapshot with "Spear Thrust" ability to test a mace.

3. Simulate that subsystem, e.g. tactical combat. That's the most difficult step, since you'll have to emulate a real player with some form of AI. It needs to know most typical combos, so tag your abilities like "damage", "stun", "slow", e.t.c., and define combos in terms "stun->slow->damage over time->damage nuke". Things will be easier if you introduce "turns" even if your game is real-time. Take your typical combat situations and run them with your snapshots with equipment and abilities appropriate to "situation level". Note which snapshots of the same type perform better than others, and which combos resulted in a complete devastation of enemy. Take your results with grain of salt, keep in mind that it's your combat AI fighting, not a player. So you may want to use several types of said AI, and playtest your balance changes yourself.

4. The biggest problem is that players can be progamers or complete casuals, and you'll have to decide on your "homogenized target audience". You can test out challenge level by taking average leveled snapshots and average equipment for current game stage, and testing them agains current game map enemies. "Average" here depends on a target audience. Difficulty curve is not that simple, it's mostly subjective feeling, so you'll have to playtest it with your target audience.