Assuming that your problem is only forIf one wants to use braces directly also in math mode, then the followingmathcodes and delcodes must be correctly set. This is a possible waydone here:
\documentclass{article} \usepackage[T1]{fontenc} \def\displaybrace<{\bgroup\catcode`\>=2 \delcode123="266308 \delcode125="267309 \mathcode123="4266 \mathcode125="5267 \catcode123=12 \catcode125=12 } \begin{document}\pagestyle{empty} \[\displaybrace<{{x},{x,y}}>\] \end{document} Any use of a > will close the macro. Within the \displaybrace<..stuff...> we have a substitute only for the closing group delimiter with >.

If you wantSo if one wants to do other things than just display setssome collection of braces and letters, then the following just slight extension is needed, which just additionally makes < also serve as opening group character:
\documentclass{article} \usepackage[T1]{fontenc} \def\displaybrace<{\bgroup\catcode`\<=1 \catcode`\>=2 \delcode123="266308 \delcode125="267309 \mathcode123="4266 \mathcode125="5267 \catcode123=12 \catcode125=12 } \begin{document}\pagestyle{empty} \[\displaybrace<{{x},{x,y}}>\] \[\omega < \displaybrace<\left{\int_a^b e^<-x^2>\,dx\right}> <z^2 \] \end{document} 
Please ntake note how I used < and > inside \displaybrace to code the superscript.
More generally any code (be it in text or math mode) where one would have used traditionnally { and } must be written with < and >. And it will work perfectly fine if this constraint is obeyed (except if some macro is used which really requires that { and } are in their traditional habits, or and this is more probable, some macro which would be completely surprised to learn that < and > are now the opening and closing group characters).
Obviously using < and > this way makes writing inequalities inside difficultimpossible... But we could let + and - play the rôle here played by < and > if you want inequalities inside ... but no additions nor substractionsAgain + and - are used for many things so one would have to find something better... etc
... (usingfor example ^^A and ^^B for(which have character codes 1 and 2) could play the rôle here attributed to \displaybrace< argument delimiting could be a possibility, butand > and this would allow complete freedom to set-up arbitrary math formulas (perhaps your sets are defined mathematically?). But then it becomes clumsythe input inside if one needs groups therein)would not be so readable.

And withLet me explicitely points out that the above code is designed to work well with arbitrary (or the earlier version which did not even yet use\left or <\big.. as a substitue for the group opening character) the following is possible:like constructions.
\[\displaybrace<\Biggl{\biggl{x\biggr},{x,y}\Biggr}>\] 