Return to Answer

sed ' :a $!{ # slurp the whole input into the pattern space N; ba } # using _ as an escape character to escape { as _l and # } as _r below. So escape itself as _u first: s/_/_u/g :b # process the \SomeStyle{...} that contain no {}: s/\\SomeStyle{\([^{}]*\)}/\1/g; tb # replace inner {...} to _l..._r and loop: s/{\([^{}]*\)}/_l\1_r/g; tb # undo escaping: s/_l/{/g; s/_r/}/g; s/_u/_/g' file.tex 

To account for braces escaped as \{ (and \ escaped as \\)¹, you can use the now standard -E option (preferable to the GNU-specific -r) to switch to extended regular expressions that have a | alternation operator, though note that { also becomes a regexp operator then and needs to be escaped when outside [...], and grouping changes from \(...\) to (...):

sed ' :a $!{ # slurp the whole input into the pattern space N; ba } # using _ as an escape character to escape { as _l and # } as _r below. So escape itself as _u first: s/_/_u/g :b # process the \SomeStyle{...}s that contain no unescaped {}: s/\\SomeStyle{\([^{}]*\)}/\1/g; tb # replace inner {...} to _l..._r and loop: s/{\([^{}]*\)}/_l\1_r/g; tb # undo escaping: s/_l/{/g; s/_r/}/g; s/_u/_/g' file.tex 

To account for braces escaped as \{ (and \ escaped as \\)¹, you can use the now standard -E option (preferable to the GNU-specific -r) to switch to extended regular expressions that have a | alternation operator, though note that { also becomes a regexp operator then and needs to be escaped when outside [...], and grouping+capturing changes from \(...\) to (...):

(add -i option to edit the file in-place).

Where we replace [^{}]* with (?:\\.|[^{}\\])* to match \anycharacter (including \\, \{ and \} which we care about here) in addition to characters other than \, {, and }. (?:...) is the non-capturing form of (...).

(add -i option to edit the file in-place).

To account for braces escaped as { (and \ escaped as \)¹, you can use the now standard -E option to switch to extended regular expressions that have a | alternation operator, though note that { also becomes a regexp operator then and needs to be escaped when outside [...], and grouping changes from \(...\) to (...):

^{¹ still ignoring the possibility that there may be \\SomeStyle{something}, not handling comments of \verb... Covering for those and do a full TeX tokenising would be possible but may not be worth the effort depending on your actual input.}

To account for braces escaped as \{ (and \ escaped as \\)¹, you can use the now standard -E option (preferable to the GNU-specific -r) to switch to extended regular expressions that have a | alternation operator, though note that { also becomes a regexp operator then and needs to be escaped when outside [...], and grouping changes from \(...\) to (...):

^{¹ still ignoring the possibility that there may be \\SomeStyle{something}, not handling comments or \verb|...|... Covering for those and do a full TeX tokenising would be possible but may not be worth the effort depending on your actual input.}