Assuming that your problem is only for math, then the following is a possible way:

 \documentclass{article}
 \usepackage[T1]{fontenc}

 \def\displaybrace<{\bgroup\catcode`\>=2 \delcode123="266308
 \delcode125="267309 \mathcode123="4266 \mathcode125="5267
 \catcode123=12 \catcode125=12 }
 
 \begin{document}\pagestyle{empty}
 
 \[\displaybrace<{{x},{x,y}}>\]
 
 \end{document}

Any use of a `>` will close the macro.

![displayed braces][1]

If you want to do other things than just display sets, then the following just slight extension is needed:

 \documentclass{article}
 \usepackage[T1]{fontenc}
 
 \def\displaybrace<{\bgroup\catcode`\<=1 \catcode`\>=2 
 \delcode123="266308
 \delcode125="267309 
 \mathcode123="4266 \mathcode125="5267
 \catcode123=12 \catcode125=12 }
 
 \begin{document}\pagestyle{empty}
 
 
 \[\displaybrace<{{x},{x,y}}>\]
 
 
 \[\omega < \displaybrace<\left{\int_a^b e^<-x^2>\,dx\right}> <z^2 \]
 
 \end{document}

Please note how I used `<` and `>` inside `\displaybrace`. Obviously this makes writing inequalities inside difficult... But we could let `+` and `-` play the rôle here played by `<` and `>` if you want inequalities inside ... but no additions nor substractions... etc... (using `^^A` and `^^B` for the `\displaybrace` argument delimiting could be a possibility, but then it becomes clumsy input inside if one needs groups therein).

![output][2]




 [1]: https://i.sstatic.net/SFfiQ.png
 [2]: https://i.sstatic.net/yK8h6.png