Thermal stress causes the breaking of rocks caused by heating/cooling. You may choose deserts (due to heat) but if you think about it, there are not many rocks present compared to caves where it is more likely.
One side of the mountain that has constant wind and rain blowing onto it, is more likely to catch what is falling than the other side leaving it dryer.