This is because Whitman doesn't consider these people as part of the working class. They have not toiled in the dirt or in the rank sweat of a factory. Politicians and rich men do not know what physical toil is nor are they relegated to survival for much of their life.