5π/8 is a 2nd-quadrant angle whose reference angle is greater than π/4. This means the tangent is negative with a magnitude greater than 1. The only reasonable choice is ...
_____
<em>Additional comment</em>
This simplifies to -(1+√2). Your calculator can verify the choice for you.