There are
(or "52 choose 3") ways of drawing any 3 cards from the deck.
There are 13 hearts in the deck, and 26 cards with a black suit. So there are
and
ways of drawing 3 hearts or 3 black cards, respectively. Then the probability of drawing 3 hearts is

and the probability of drawing 3 black cards is

All other combinations can be drawn with probability
.
Let
be the random variable for one's potential winnings from playing the game. Then

a. For a single game, one can expect to win
![E[W]=\displaystyle\sum_ww\,P(W=w)=\frac{\$50\cdot11}{850}+\frac{\$20\cdot2}{17}+\frac{\$0\cdot739}{850}=\$3](https://tex.z-dn.net/?f=E%5BW%5D%3D%5Cdisplaystyle%5Csum_ww%5C%2CP%28W%3Dw%29%3D%5Cfrac%7B%5C%2450%5Ccdot11%7D%7B850%7D%2B%5Cfrac%7B%5C%2420%5Ccdot2%7D%7B17%7D%2B%5Cfrac%7B%5C%240%5Ccdot739%7D%7B850%7D%3D%5C%243)
b. For a single game, one's winnings have a variance of
![V[W]=E[(W-E[W])^2]=E[W^2]-E[W]^2](https://tex.z-dn.net/?f=V%5BW%5D%3DE%5B%28W-E%5BW%5D%29%5E2%5D%3DE%5BW%5E2%5D-E%5BW%5D%5E2)
where
![E[W^2]=\displaystyle\sum_ww^2\,P(W=w)=\frac{\$50^2\cdot11}{850}+\frac{\$20^2\cdot2}{17}+\frac{\$0^2\cdot739}{850}=\$^2\frac{1350}{17}\approx\$^279.41](https://tex.z-dn.net/?f=E%5BW%5E2%5D%3D%5Cdisplaystyle%5Csum_ww%5E2%5C%2CP%28W%3Dw%29%3D%5Cfrac%7B%5C%2450%5E2%5Ccdot11%7D%7B850%7D%2B%5Cfrac%7B%5C%2420%5E2%5Ccdot2%7D%7B17%7D%2B%5Cfrac%7B%5C%240%5E2%5Ccdot739%7D%7B850%7D%3D%5C%24%5E2%5Cfrac%7B1350%7D%7B17%7D%5Capprox%5C%24%5E279.41)
so that
. (No, that's not a typo, variance is measured in squared units.) Standard deviation is equal to the square root of the variance, so it is approximately $8.39.
c. With a $5 buy-in, the expected value of the game would be
![E[W-\$5]=E[W]-\$5=-\$2](https://tex.z-dn.net/?f=E%5BW-%5C%245%5D%3DE%5BW%5D-%5C%245%3D-%5C%242)
i.e. a player can expect to lose $2 by playing the game (on average).
d. With the $5 cost, the variance of the winnings is the same, since the variance of a constant is 0:
![V[W-\$5]=V[W]](https://tex.z-dn.net/?f=V%5BW-%5C%245%5D%3DV%5BW%5D)
so the standard deviation is the same, roughly $8.39.
e. You shouldn't play this game because of the negative expected winnings. The odds are not in your favor.