Answer:
a) 
the smallest n that maximizes the expected payoff is n=5.
b) 4
Step-by-step explanation:
a)
The expected amount of $ won for each die would be the average of 1, 2, 3, 4 and 5 which is $3.
Let W(n) the expected money won when rolling n dice.
n =1
If the gambler rolls only one die, the expected money won would be $3 times the probability of not getting a 6, which is 5/6.
So
n=2
If the gambler rolls 2 dice, the expected money won would be $3 times the probability of not getting a 6 in any of the dice. Since the outcome of the rolling does not depend on the previous rollings, the probability is
and
n=3
Similarly, since the probability of not getting a 6 in 3 dice equals
and the formula for the expected money won with n dice would be
In the picture attached there is a plot of the values of the expected money won for n=1 to 20 (See picture)
For n=5 and n=6 we get the maximum profit expected of $6.02816=$6 rounded to the nearest integer.
Hence, the smallest n that maximizes the expected payoff is n=5.
b)
The probability that face k (k=1,2,...or 6) shows up is 1/6,
as this face can be in any of the 10 positions of the arrangement, there are 10 ways that face k can show up.
The probability that face k (k=1,2,...or 6) shows up twice is 
as this face can be in any of the 
(combinations of 10 taken 2 at a time) positions of the arrangement, there are C(10;2) ways that face k can show up twice.
The probability that face k (k=1,2,...or 6) shows up three times is 
as this face can be in any of the 
(combinations of 10 taken 2 at a time) positions of the arrangement, there are C(10;3) ways that face k can show up twice.
So, we infer that the expectation is
and the expected number of distinct dice values that show up is 4.
