The 4e DMG has some advice for handling this player type.
Although of course you're playing D&D 5e, the earlier 4th edition Dungeon Master's Guide actually has advice on handling what it calls the Instigator player type.
An instigator enjoys making things happen. She has no patience for careful planning or deliberation. She'll open an obviously trapped chest "just to see what happens." She provokes authority figures and opens dungeon doors to bring more monsters into an already difficult fight. The instigator loves the vicarious thrill of taking enormous risks and sometimes just making bad choices.
The instigator can be disruptive, but she can also be a lot of fun for the othe players. Things rarely grind to a halt with an instigator in the group, and the stories that get retold after the game session often revolve around whatever crazy thing the instigator did this week.
The 4e DMG gives some advice on handling this sort of player.
Don't allow PK/TPK: The most important is to stop the Instigator from attacking the party, or getting the party killed. The DMG doesn't specify exactly how the DM should do this.
However, it also recommends engaging the Instigator to create scenarios that they find fun, which should satisfy the player's need for chaos. It gives three main pieces of advice:
Include objects and encounters that invite experimentation: Intentionally include encounters where the player character can harmlessly mess around without de-railing the adventure or harming the rest of the party. I like to think of this as Marge baking a second cake for Homer to ruin.
Let their actions put the PCs in a tight spot: Don't just say "no" to the player, but have their actions lead to some kind of combat encounter or other interesting scenario for the party. My advice is that this should be a brief and dramatic occurence but not seem like a punishment for the party.
Include encounters with Instigator NPCs: Have the player encounter NPCs who pull the exact same stunts back at them. You stole from a merchant? Turns out he pays dues to the local thieves' guild, who try to rob the thief back to teach him a lesson. Give the player a taste of their own medicine.
My own suggestion
My own advice might be that if the player is hoping to pull silly stunts for their own amusement, that their come-uppance would be better as more of a "punchline" than a "punishment". This will give the player the amusement they hoped for without hindering the adventure or playstyle. I'd quickly narrate the results of failed attempts to cause trouble, before moving on to continue the adventure. Examples:
- You try to loot the merchant's purse, only to notice that he's staring right at you the whole time. He picks you up and dumps you into a horse trough outside.
- You are caught almost immediately trying to steal the item in plain sight. A mass of guards immediately spot you and cart you off to jail. A week passes before they decide to let you free.
By making it amusing, you allow the PC to fail without taking it too harshly. By making it quick, you avoid allowing the PC to use their chaotic "my guy" antics to dominate the game, cause the minimum hassle for the other players, avoid upsetting the Instigator with drawn-out punishments, and continue the game as intended.