Typical approach is to use `perl`'s recursive regex capability:

```
perl -0777 -pe 's/\\SomeStyle(\{((?:(?1)|[^{}])*)\})/$2/gs' file.tex
```

(add `-i` option to edit the file `i`n-place).

Above `(?1)` is like inserting the regexp in the first pair of `(...)`, so `(\{((?:(?1)|[^{}])*)\})` at that point.

If the `\SomeStyle{...}`s may be nested as in:

```
\SomeStyle{\otherstyle{this is
the \SomeStyle{\textit{nested part} some} more text...}}
```

To be changed to:

```
\otherstyle{this is
the \textit{nested part} some more text...}
```

Then change it to:

```
perl -0777 -pe '
 while(s/\\SomeStyle(\{((?:(?1)|[^{}])*)\})/$2/gs){}' file.tex
```

Which will repeat the process, replacing the outer ones first until no match is found.

With `sed` assuming and implementation where the whole input can fit in the patter, one approach (also handling nested ones) could be:

```
sed '
 :a
 $!{
 # slurp the whole input into the pattern space
 N;ba
 }
 # escape _, <, > as _u, _l, _g respectively so we can user
 # < and > as replacements below:
 s/_/_u/g;s/</_l/g;s/>/_/g
 :b
 # process the \SomeStyle{...} that contain no {}:
 s/\\SomeStyle{\([^{}]*\)}/\1/g;tb
 # replace inner {...} to <...> and loop:
 s/{\([^{}]*\)}/<\1>/g;tb
 y/<>/{}/
 s/_g/>/g;s/_l/</g;s/_u/_/g' file.tex
```

Some `sed` implementations have copied perl's `-i` for in-place editing but beware that in some (FreeBSD's and derivatives), you need `-i ''` for in-place editing without backup of the original. `-i.back` would work in all implementations that have a `-i` (and in perl) and save the original as `file.tex.back`.