The dimensions of the square base after cutting the corners will be (40-2x). The depth of the box will be x, so its volume is given by
V = x(40-2x)²
You can differentiate this to get
V' = 12x² -320x -1600
Setting this to zero and factoring gives
(3x-20)(x-20) = 0
The appropriate choice of solutions is
x = 20/3 = 6 2/3
The dimensions of the box of maximum volume are
26 2/3 in square by 6 2/3 in deep
The maximum volume is
(80/3 in)²(20/3 in) = 4740 20/27 in³
V = x(40-2x)²
You can differentiate this to get
V' = 12x² -320x -1600
Setting this to zero and factoring gives
(3x-20)(x-20) = 0
The appropriate choice of solutions is
x = 20/3 = 6 2/3
The dimensions of the box of maximum volume are
26 2/3 in square by 6 2/3 in deep
The maximum volume is
(80/3 in)²(20/3 in) = 4740 20/27 in³