Just don't use \ensuremath: it adds a level of grouping that makes impossible for \IfSubStr to find \correctanswer; indeed the test
\IfSubStr{1{2}3}{2}{true}{false}
returns "false". So write
\newcommand{\mymathtext}[1]{\mytext{$#1$}}
Please, note that your code adds plenty of spurious spaces:
\newcommand{\mytext}[1]{% Text : \noexpandarg \IfSubStr{#1}{\correctanswer}{% \textbf{\boldmath #1}% }{% #1% } }
I would rewrite it as
\newcommand{\mytext}[1]{% Text : % be sure we want a space \noexpandarg \IfSubStr{#1}{\correctanswer} {\textbf{\boldmath #1}} {#1}% }
But probably the best thing is to define a *-variant:
\makeatletter \newcommand{\mytext}{\@ifstar\mytext@s\mytext@n} \newcommand{\mytext@s}[1]{\textbf{\boldmath #1}} \newcommand{\mytext@n}[1]{#1} \makeatother
and say \mytext{Yes} or \mytext*{No} (correct answer); for math just add $: \mytext{$\frac{1}{2}$} or \mytext*{$\frac{1}{4}$}
This is the standard way to define a command with a *-variant; since we need to use @ commands, the code must be enclosed between \makeatletter and \makeatother.
The command \mytext is defined to peek at what follows it with \@ifstar; if a * follows, we execute \mytext@s, otherwise \mytext@n, that we define next just normally: \mytext@s{x} does \textbf{\boldmath x}, while \mytext@n{x} does simply x. Indeed TeX will substitute \mytext* with \mytext@s, so it will next see the argument to pass to this command, and similarly without a *.
In order to arrange things with \ifsolution, you can say
\newif\ifsolution \makeatletter \newcommand{\mytext}{\@ifstar\mytext@s\mytext@n} \newcommand{\mytext@s}[1]{% \begingroup \ifsolution\bfseries\boldmath\fi #1% \endgroup} \newcommand{\mytext@s}[1]{#1} \makeatletter
so that the \bfseries\boldmath commands will be issued only when \ifsolution is set to true.
Simpler version
A simpler definition might be:
\usepackage{xparse} \newif\ifsolution \NewDocumentCommand{\answer}{st{+}m} {\begingroup \ifsolution\IfBooleanT{#1}{\bfseries\boldmath}\fi \IfBooleanT{#2}{$}#3\IfBooleanT{#2}{$}% \endgroup} \answer{Yes} \answer*{No} \answer+{\frac{1}{2}} \answer*+{\frac{1}{4}} \solutiontrue \answer{Yes} \answer*{No} \answer+{\frac{1}{2}} \answer*+{\frac{1}{4}}
You'll get
Yes No
1/2 1/4
Yes No
1/2 1/4
with the fractions in math mode, of course. The presence of + implies a formula, as you can see.
EDIT
Assuming that you have a \answerformat macro that formats answers, say
\newcommand{\answerformat}[1]{\mbox{\checkbox\qquad{#1}}\\}
(where \checkbox prints a box), you can use a syntax such as
\answers{Yes}*{No}+{\frac{1}{2}}+{\frac{1}{4}}
or even
\answers[3]{Yes}{No}*+{\frac{1}{2}}
that is, preceding the correct answer by * and math answers by +; the optional argument tells how many choices you have (default 4):
\usepackage{xparse} \NewDocumentCommand{\answer}{st{+}m} {\answerformat{% \ifsolution\IfBooleanT{#1}{\bfseries\boldmath}\fi \IfBooleanT{#2}{$}#3\IfBooleanT{#2}{$}} \ifnum\value{anscount}<\answernumber \stepcounter{anscount}\expandafter\answer \fi} \NewDocumentCommand{\answers}{O{4}} {\setcounter{anscount}{1}\chardef\answernumber=#1\relax \answer} \newcounter{anscount}
\choicecommand chich takes four arguments, with one (or more) containing\correctanswer, which allows me to use the same document for the test and the correction, by switching between\solutionstrueand\solutionsfalse.\correctanswer{\mytext{Bar}}, defining it either as\newcommand{\correctanswer}[1]{#1}or\newcommand{\correctanswer}[1]{\textbf{#1}}depending on\ifsolutions? Usingxstringhere seems like an overkill.