#It overpowers disadvantage stacking, and underpowers other spells/abilities

While your proposed system addresses the realism factor to a degree, mechanically it changes the game in a way that can be frustrating, as it completely negates the balancing effect certain spells and abilities have on advantage/disadvantage.

For example, suppose your Enemy has Blur and Foresight up and is able to poison you. Even if your ally manages to restrain them and someone uses the Help action, you're still going to be at disadvantage, rendering mechanics that negate disadvantage like help/flanking useless unless you can stack enough of them to even hit neutral. Stacking multiple effects becomes an arms race between stacking disadvantage to guarantee it, or to gaining enough stacked advantage to simply negate it.

The choice to keep it limited to cancellation is easier for tracking, prevents overpowered stacking, and keeps things running smoother overall. It's not perfect, but for the majority of cases it does well at approximating in a way that is simple and fun. As with many edge cases, specific DM ruling in a situation can often be better than broad rule changes.

It overpowers disadvantage stacking, and underpowers other spells/abilities

While your proposed system addresses the realism factor to a degree, mechanically it changes the game in a way that can be frustrating, as it completely negates the balancing effect certain spells and abilities have on advantage/disadvantage.

For example, suppose your Enemy has Blur and Foresight up and is able to poison you. Even if your ally manages to restrain them and someone uses the Help action, you're still going to be at disadvantage, rendering mechanics that negate disadvantage like help/flanking useless unless you can stack enough of them to even hit neutral. Stacking multiple effects becomes an arms race between stacking disadvantage to guarantee it, or to gaining enough stacked advantage to simply negate it.

The choice to keep it limited to cancellation is easier for tracking, prevents overpowered stacking, and keeps things running smoother overall. It's not perfect, but for the majority of cases it does well at approximating in a way that is simple and fun. As with many edge cases, specific DM ruling in a situation can often be better than broad rule changes.

#The proposed change can lead to similar situations

You've provided an example of a situation that is made rather odd by the current implementation. But equally odd situations are likely occur with the change you suggested as well.

For example, suppose you hide in the fog while your ally restrains the enemy outside of it. Being blind cancels with being unseen, but because your ally is restraining them, somehow your attacks are advantageous. Suppose then that the enemy has Blur up to equalize it again. Then you just have someone use the Help action to cancel that out, and then you're back to advantage again.

Your proposed solution solves one odd situation, but introduces even more in turn.

#It also overpowers advantage/disadvantage stacking

As shown above, stacking multiple effects becomes an arms race to guarantee advantage/disadvantage. If you have a spell like Blur up, by RAW at the very least it should always negate advantage against you, even if they don't have disadvantage. You're still getting something out of it. With this proposition however, they may still have advantage on you if they stack enough, making your Blur a waste of concentration.

The choice to keep it limited to cancellation is easier for tracking, prevents overpowered stacking, and keeps things running smoother overall. It's not perfect, but for the majority of cases it does well at approximating in a way that is simple and fun. As with many edge cases, specific DM ruling in a situation can often be better than broad rule changes.

#It overpowers disadvantage stacking, and underpowers other spells/abilities

While your proposed system addresses the realism factor to a degree, mechanically it changes the game in a way that can be frustrating, as it completely negates the balancing effect certain spells and abilities have on advantage/disadvantage.

For example, suppose your Enemy has Blur and Foresight up and is able to poison you. Even if your ally manages to restrain them and someone uses the Help action, you're still going to be at disadvantage, rendering mechanics that negate disadvantage like help/flanking useless unless you can stack enough of them to even hit neutral. Stacking multiple effects becomes an arms race between stacking disadvantage to guarantee it, or to gaining enough stacked advantage to simply negate it.

The choice to keep it limited to cancellation is easier for tracking, prevents overpowered stacking, and keeps things running smoother overall. It's not perfect, but for the majority of cases it does well at approximating in a way that is simple and fun. As with many edge cases, specific DM ruling in a situation can often be better than broad rule changes.

#The proposed change can lead to similar situations

You've provided an example of a situation that is made rather odd by the current implementation. But equally odd situations are likely occur with the change you suggested as well.

For example, suppose you hide in the fog while your ally restrains the enemy outside of it. Being blind cancels with being unseen, but because your ally is restraining them, somehow your attacks are advantageous. Suppose then that the enemy has Blur up to equalize it. Then you just have someone use the Help action to cancel that out, and then you're back to advantage again.

Your proposed solution solves one odd situation, but introduces even more in turn.

#It also overpowers advantage/disadvantage stacking

As shown above, stacking multiple effects becomes an arms race to guarantee advantage/disadvantage. If you have a spell like Blur up, by RAW at the very least it should always negate advantage against you, even if they don't have disadvantage. You're still getting something out of it. With this proposition however, they may still have advantage on you if they stack enough, making your Blur a waste of concentration.

The choice to keep it limited to cancellation is easier for tracking, prevents overpowered stacking, and keeps things running smoother overall.

#The proposed change can lead to similar situations

You've provided an example of a situation that is made rather odd by the current implementation. But equally odd situations are likely occur with the change you suggested as well.

For example, suppose you hide in the fog while your ally restrains the enemy outside of it. Being blind cancels with being unseen, but because your ally is restraining them, somehow your attacks are advantageous. Suppose then that the enemy has Blur up to equalize it again. Then you just have someone use the Help action to cancel that out, and then you're back to advantage again.

Your proposed solution solves one odd situation, but introduces even more in turn.

#It also overpowers advantage/disadvantage stacking

As shown above, stacking multiple effects becomes an arms race to guarantee advantage/disadvantage. If you have a spell like Blur up, by RAW at the very least it should always negate advantage against you, even if they don't have disadvantage. You're still getting something out of it. With this proposition however, they may still have advantage on you if they stack enough, making your Blur a waste of concentration.

The choice to keep it limited to cancellation is easier for tracking, prevents overpowered stacking, and keeps things running smoother overall. It's not perfect, but for the majority of cases it does well at approximating in a way that is simple and fun. As with many edge cases, specific DM ruling in a situation can often be better than broad rule changes.