You have the correct answer. The blue plane is basically a knife that cuts the pyramid into two pieces. The top portion is a smaller similar pyramid (same shape, smaller size). The bottom portion is a pyramidal frustum.
Where the cut happens in the blue plane, a square forms. This square is the base of the upper pyramid and the roof of the frustum.
We can think of the blue stuff like water. Then allow that water to freeze to form a block of ice. Then carefully slide the block of ice upward and there should be a square hole in the center. So this is another way to see how we have a square cross section. The square cross section is due to the pyramid's base being a square. This means that if we had say a hexagonal base, then the cross section would be hexagonal.
All of this only applies if the cutting plane is parallel to the base of the pyramid.
