PEMDAS
5X10 = 50
102-50= 52
Answer = 52
It should be 5/13 because the opposite angle from D is 12, which you would use if you are looking for sine. But, since you are looking for cosine, it is adjacent over hypotenuse. Therefore leaving you with 5/13, since your hypotenuse is always your longest side and the adjacent is the other side from the one opposite of the angle.
You have the correct chosen