The dimensions of the square base after cutting the corners will be (40-2x). The depth of the box will be x, so its volume is given by V = x(40-2x)²
You can differentiate this to get V' = 12x² -320x -1600 Setting this to zero and factoring gives (3x-20)(x-20) = 0 The appropriate choice of solutions is x = 20/3 = 6 2/3
The dimensions of the box of maximum volume are 26 2/3 in square by 6 2/3 in deep
The maximum volume is (80/3 in)²(20/3 in) = 4740 20/27 in³