Find the equation of of the plane that contains all three points'

Start with figure 1. Work your way left to right, top to bottom til you get to figure 4. This is the basic process to turn a flat piece of cardboard into a 3D folded box.

The dashed lines in figure 1 are x inches long.

Figure 2 is the result of cutting out the dashed lines you see in figure 1.

When you fold along the dashed lines in figure 2, you'll end up with something similar to figure 3.

Figure 4 is the completed open-top-box. In other words, this box does not have a lid.

The green portions in figures 3 and 4 indicate where the flaps used to be in reference to figure 2.

-------------------

Since those dashed lines in figure 1 were x inches long, this means that the distance from A to B is 11-x-x = 11-2x inches long.

This makes segment BD to be 8-2x inches long. We are subtracting off 2 copies of x each time for each dimension.

------------------

length = AB = 11-2x

width = BD = 8-2x

Since we want the length to be positive, this means 11-2x > 0 which solves to x < 5.5

Since we want the width to be positive, this means 8-2x > 0 which solves to x < 4

Overall, we must make x < 4 to make both inequalities 11-2x > 0 and 8-2x > 0 to be true at the same time.

Furthermore, x is some length itself, so it cannot be negative either. Therefore x > 0 or 0 < x

Combine 0 < x and x < 4 to get 0 < x < 4

The restriction 0 < x < 4 will be used later.

------------------

For now, let's compute the volume.

Volume = Length*Width*Height

Volume = (AB)*(BD)*x

Volume = (11-2x)*(8-2x)*x

Volume = (88-22x-16x+4x^2)*x

Volume = (88-38x+4x^2)*x

Volume = 88x-38x^2+4x^3

This is the algebraic way of representing the volume in terms of the cut length x

We want this volume to be 45 cubic inches. We set 88x-38x^2+4x^3 equal to 45 and solve for x

88x-38x^2+4x^3 = 45

4x^3-38x^2+88x-45 = 0

At this point we try to factor by grouping or we turn to a calculator. Cubics are often very difficult to solve by hand, so I recommend using a calculator (unless you can quickly spot how to factor by grouping, but I don't see a quick way of doing that). Note: based on the answer choices that aren't whole numbers (assuming choice C is a non-answer), this heavily implies we do not have rational roots, and therefore factoring over the rationals will be impossible. In other words, the answer choices tell us to immediately use a graphing calculator to find the roots.

Using geogebra (see figure 5), I find the three approximate roots to be

x = 0.716 (shown as point A)

x = 2.5 (point B)

x = 6.284 (point C)

Recall that earlier we made 0 < x < 4. The first two roots satisfy this inequality, but x = 6.284 is outside this range. So we ignore it. It looks like your teacher put 6.284 as a trick answer in choice A.