You need to subtract the two cutouts on each side, so the side of the box would be 24- 2x. X would then become the height of the box once folded.
The volume is found by multiplying the 4 dimensions:
The volume becomes x(24-2x)^2
Now take the derivative of the formula to solve for x
Dv/dx= (24-2x)^2 + x^2(24-2x)*-2
= 24-2x(24-2x-4x)=0
X =12 or 4
24-2x = 24-2(4) = 24-8=16
Volume = 16 x 16 x 4 = 1024 cubic inches.