We're adding two vectors here. The first is 300 Newtons to the right, which we can write as (300, 0), meaning 300 to the right, 0 up.
The second is 300 at let's say a 45 degree angle down. For the components we have an isosceles right triangle with hypotenuse 300, so the components are both magnitude 300/√2 = 150√2. So we can write this vector (150√2, -150√2), the negative sign because it points down in the y direction.
Adding is componentwise. The resulting force is (300+150√2, -150√2).
That has square magnitude
r² = (300+150√2)² + (-150√2)² = 150² ( (2+√2)² + (√2)² )
= 150²( (6 + 4√2) + 2)
= 300²(2+√2)
so
r = 300 √(2+√2) Newtons
That's the answer; I'm not sure if your class expects a calculator approximation, which is 554.3 Newtons.
