If x is the length of the sides of the squares, then the height of the box is also x. The length and width of the base are 10−2x and 20−2x. The area of the base is the length times the width.
96 = (10 − 2x) (20 − 2x)
96 = 200 − 20x − 40x + 4x²
0 = 4x² − 60x + 104
0 = x² − 15x + 26
0 = (x − 2) (x − 13)
x = 2 or 13
Since x < 5, x = 2.
So the length of the sides of the squares is 2 cm.