We need to figure out how much of the total area of the picture frame is usable. To do this, we need to figure out what is taken up by the frame and the mat.
The frame is 36 inches on a side. We know the frame is 2 inches. We know that the the mat is 5 inches.
This means that on each 36 inch side, 4 inches are taken up by the frame (2 each side) and 10 inches by the mat (again, 5 each side). So the total length left is 36 - 4 - 10 = 32 - 10 = 22.
The maximum area of the photograph is therefore 22 x 22 = 484 square inches.