This sort of question is answered easily by a graphing calculator. The square cut from each corner should be 1.962 inches on each side.
_____
After creating a fold-up flap of x inches in width, the base of the box will be
(10 - 2x) by (15 - 2x)
and the depth of the box will be the width of the fold-up flap: x.
Then the volume of the box is
v = x(10 -2x)(15 -2x) = 150x -50x^2 +4x^3
The derivative of the volume will be zero at the maximum volume.
0 = dv/dx = 150 -100x +12x^2
This has roots at
x = (100 ±√(100² - 4(12)(150)))/(2·12)
x = (100 ± √2800)/24 = (25 ± 5√7)/6
Only the smaller of these solutions gives a maximum volume.
You should cut (5/6)(5-√7) ≈ 1.962 inches to obtain the greatest volume.
_____
After creating a fold-up flap of x inches in width, the base of the box will be
(10 - 2x) by (15 - 2x)
and the depth of the box will be the width of the fold-up flap: x.
Then the volume of the box is
v = x(10 -2x)(15 -2x) = 150x -50x^2 +4x^3
The derivative of the volume will be zero at the maximum volume.
0 = dv/dx = 150 -100x +12x^2
This has roots at
x = (100 ±√(100² - 4(12)(150)))/(2·12)
x = (100 ± √2800)/24 = (25 ± 5√7)/6
Only the smaller of these solutions gives a maximum volume.
You should cut (5/6)(5-√7) ≈ 1.962 inches to obtain the greatest volume.