about the second one... well, is a "fait accompli" that using the pythagorean theorem, if x = 8 and y = 5, the hypotenuse must be √(8² + 5²) = √(89), which is neither of those choices.
5, 8, 13 are no dice, namely 5² + 8² ≠ 13
25, 64, 17 is are no dice too, because 25² + 17² ≠ 64²
however, 5,12 and 13 are indeed a pythagorean triple
also is 39, 80, 89.
when looking for a pythagorean triple, recall that c² = a² + b².
so the longest leg is the sum of the square of the small ones.
so what you'd do is, check the small legs, square them, add them up, if they're indeed a pythagorean triple, they "must" add up to the longest leg.