Gandhiji says this because he tried to become an English gentleman by adopting the British culture and his ambition grew like the family of the recluse. He wasted time, money, and energy in his endeavors of becoming an English gentleman, which were not related to his studies for which he was in England.
