The speaker would rather tear down the wall and get closer to his neighbor. He sees no reason for the wall as their apple and pine trees aren't going to bother one another and there are no cattle to hold into either property. The neighbor is the one who repeatedly says "good fences make good neighbours."