Plantation owners turned to the enslaved Africans as a labor force because they were not paying them for their work, instead they were just providing shelter and food for them. Also, they had total control over them because they owned them, thus were able to make them work as much as they wanted and the slaves couldn't do anything about it.
men have jobs and responsibilities they might not be able to handle and they have limits in their responsibility and could be as liable to harm as much as women.