They didnt like how he thought of federal government. They didnt want him to be involved in changing the south and its government/society during Reconstruction.
This is false. The gardens were considered one of the seven wonders of the world; little evidence is found of the garden today believed to have been destroyed sometime in 1 AD.