Answer: Approximately 3.42857142857142 inches
Round this value however you need to. For instance, if you need to round to 2 decimal places, then the answer would be 3.43 inches.
====================================================
How I got this answer:
A square has side lengths that are the same value and it has four right angles. For example, a square could have all four sides equal to 7 inches. We want this square to fit snugly inside a right triangle. Put another way, we want this square as big as possible without it overflowing outside the triangle.
If one vertex of the square is at (0,0) then the opposite vertex will be on the line y = x meaning this opposite vertex's location is (x,x). The x refers to the side length of the square. Replace x with any number. Going back to the previous example, if x = 7 then the square has two opposite vertices at (0,0) and (x,x) = (7,7), so this square has side length of 7. We will use the equation y = x later on (see figure 2 in the attached images below). I marked (0,0) as point A and (x,x) as point D in figure 3.
We will also use the equation of the line that goes through the two points B(0,6) and C(8,0) as shown in figure 3. The line that goes through points B and C is the line that forms the hypotenuse of the right triangle. If we can find the equation of this line, then we can use algebra to find the intersection point D (see figure 3)
It turns out that the equation of the line through B and C is y = (-3/4)x + 6. Check out figure 1 in the attached images below to see how I got that.
Then head over to figure 2 to see how I'm using the two equations to solve for x. The exact solution is the fraction x = 24/7 which approximates to x = 3.42857142857142 and you can round this however you want for the final answer.
