As the agriculture's are used to villages life and have to take care of the larger tracks of the lands and they thus live close by as compared to the semi-permanent horticulturalists.
As horticulture is not a permanent type of occupation and they don't need to bee close to the farms and required less maintenance.