If i correctly understood the situation, then the possible occurrences are respecting one of the three patterns
- XXX, three balls of same color, and the win is 10,
- XXY, possibly in an other order, and the win is 5,
- XYZ, and the win is 2.
(Different letters mark different colors.)
Totally there are 3+4+5 = 12 balls. Let us number them, 1,2,3,...,11,12. 1,2,3 are W(hite), 4,5,6,7 are R(ed), the remaining 8,9,10,11,12 B(lack). There are 12.11.10 possibilities to extract them, getting thus ordered extractions.
How many times do we get the pattern XXX? There are $$3\cdot 2\cdot 1+4\cdot 3\cdot 2+5\cdot 4\cdot 3=6+24+60=90$$ cases.
How many times do we get the pattern XXY or XYX or YXX? Three times $XXY$. We split the counting depending on the choice of the two colors $X,Y$. There are $$ \begin{aligned} &3(3\cdot 2\cdot (4+5)+4\cdot 3\cdot (3+5)+5\cdot 4\cdot (3+4)) \\ &\qquad =3(54 + 96 + 140) \\ &\qquad= 3\cdot 290 \\ &\qquad= 870 \end{aligned} $$ cases. (One can get the number of this "rather complicated countinq" after having the two simpler numbers, "excluding" them from the total.)
Finally, XYZ, we permute to get WRB, there are $3!=6$ permutations, so there are $$3!\cdot 3\cdot 4\cdot 5=6\cdot 60=360$$ cases.
The corresponding probabilities are thus $$ \frac{ 90}{1320}\ ,\qquad \frac{870}{1320}\ ,\qquad \frac{360}{1320}\ . $$ The expected win is $$ 10\cdot \frac{ 90}{1320}+ 5\cdot \frac{870}{1320}+ 2\cdot \frac{360}{1320} = \frac {199}{44} \approx 4.52272727272727\dots\ . $$
Computer check, since it is easy, using sage here:
balls = [0..11] colors = 'WWWRRRRBBBBB' X, XY, XYZ = 0, 0, 0 A = Arrangements( balls, 3 ) for a in A: extracted_colors = len( Set( [ colors[a[k]] for k in [0,1,2] ] ) ) if extracted_colors == 1: X += 1 elif extracted_colors == 2: XY += 1 elif extracted_colors == 3: XYZ += 1 print "Probabilities:", X/len(A), XY/len(A), XYZ/len(A) print "Expected win:", 10*X/len(A) + 5*XY/len(A) + 2* XYZ/len(A)
Results:
Probabilities: 3/44 29/44 3/11 Expected win: 199/44
