This is because Ben is an old gardener and is often times stubborn. When he first meets Mary he tells her "<span>We're neither of us good lookin' an' we're both of us as sour as we look". This is basically saying that they are both unattractive and disagreeable.