The rational root theorem states that the rational roots of a polynomial can only be in the form p/q, where p divides the constant term, and q divides the leading term.
In your case, both the leading term 5 and the constant term 11 are primes, so their only divisors are 1 and themselves.
So, the only feasible solutions are
For the record, in this case, none of the feasible solutions are actually a root of the polynomial.
As the exercise says, the triangles are similar. So, we can set up proportions between correspondent sides.
In order to solve for x we can set up the proportion between the horizontal and vertical sides:
Solving this proportion for x implies
Now you can solve for m and p using the pythagorean theorem, because both triangles are right:
Then, we know that the hypothenuse of the big triangle is m+p, so we have
which implies