The volume as a function of the location of that vertex is
... v(x, y, z) = x·y·z = x·y·(100-x²-y²)
This function is symmetrical in x and y, so will be a maximum when x=y. That is, you wish to maximize the function
... v(x) = x²(100 -2x²) = 2x²(50-x²)
This is a quadratic in x² that has zeros at x²=0 and x²=50. It will have a maximum halfway between those zeros, at x²=25. That maximum volume is
... v(5) = 2·25·(50-25) = 1250
The maximum volume of the box is 1250 cubic units.
