Typical approach is to use `perl`'s recursive regex capability:

```
perl -0777 -pe 's/\\SomeStyle(\{((?:(?1)|[^{}])*)\})/$2/gs' file.tex
```

Or if you have to account for braces escaped as `\{` (and `\` escaped as `\\`)¹

```
perl -0777 -pe 's/\\SomeStyle(\{((?:(?1)|\\.|[^\\{}])*+)\})/$2/gs' file.tex
```

(add `-i` option to edit the file `i`n-place).

Above `(?1)` is like inserting the regexp in the first pair of `(...)`, so `(\{((?:(?1)|\\.|[^\\{}])*+)\})` at that point.

If the `\SomeStyle{...}`s may be nested as in:

```
\SomeStyle{\otherstyle{this is
the \SomeStyle{\textit{nested part} some} more text...}}
```

To be changed to:

```
\otherstyle{this is
the \textit{nested part} some more text...}
```

Then change it to:

```
perl -0777 -pe '
 while(s/\\SomeStyle(\{((?:(?1)|\\.|[^\\{}])*+)\}){}' file.tex
```

Which will repeat the process, replacing the outer ones first until no match is found.

To do it for arbitrary style and files:

```
#! /bin/sh -
[ "$#" -ge 2 ] || {
 printf>&2 '%s\n' "Usage: $0 <style> <file> [<file>...]"
 exit 1
}
style=$1; shift
exec perl -0777 -spi -e '
 while(s/\\\Q$style\E(\{((?:(?1)|\\.|[^\\{}])*+)\})/$2/gs) {}
 ' -- -style="$style" -- "$@"
```

With `sed`, assuming an implementation where the whole input can fit in the pattern space, one approach (also handling nested ones, starting with the inner ones in this case) could be:

```
sed '
 :a
 $!{
 # slurp the whole input into the pattern space
 N; ba
 }
 # using _ as an escape character to escape { as _l and
 # } as _r below. So escape itself as _u first:
 s/_/_u/g
 :b
 # process the \SomeStyle{...} that contain no {}:
 s/\\SomeStyle{\([^{}]*\)}/\1/g; tb
 # replace inner {...} to _l..._r and loop:
 s/{\([^{}]*\)}/_l\1_r/g; tb
 # undo escaping:
 s/_l/{/g; s/_r/}/g; s/_u/_/g' file.tex
```

(same kind of approach as at https://unix.stackexchange.com/q/503755 and a few others here).

Some `sed` implementations have copied perl's `-i` for in-place editing but beware that in some (FreeBSD's and derivatives), you need `-i ''` for in-place editing without backup of the original. `-i.back` would work in all implementations that have a `-i` (and in perl) and save the original as `file.tex.back`.

Your `sed` seems to be GNU `sed` as you're using quite a few GNUisms and GNU `sed` does support `-i` à la `perl` and AFAIK doesn't have a limit other than available memory on the size of the pattern space.

To account for braces escaped as \{ (and \ escaped as \\)¹, you can use the now standard `-E` option to switch to extended regular expressions that have a `|` alternation operator, though note that `{` also becomes a regexp operator then and needs to be escaped when outside `[...]`, and grouping changes from `\(...\)` to `(...)`:

```
sed -E '
 :a
 $!{
 # slurp the whole input into the pattern space
 N; ba
 }
 # using _ as an escape character to escape { as _l and
 # } as _r below. So escape itself as _u first:
 s/_/_u/g
 :b
 # process the \SomeStyle{...} that contain no {}:
 s/\\SomeStyle\{((\\.|[^{}\\])*)\}/\1/g; tb
 # replace inner {...} to _l..._r and loop:
 s/\{((\\.|[^{}\\])*)\}/_l\1_r/g; tb
 # undo escaping:
 s/_l/{/g; s/_r/}/g; s/_u/_/g' file.tex
```

---
<sup>¹ still ignoring the possibility that there may be `\\SomeStyle{something}`, not handling comments of `\verb`... Covering for those and do a full TeX tokenising would be possible but may not be worth the effort depending on your actual input.</sup>