When Caesar says "He is a dreamer, let us leave him" (1.2.26), he is referring to the soothsayer. Furthermore, the soothsayers tells him "beware the ides of March", and Caesar brushes it off and basically calls the soothsayer insane. So, he is basically saying "He is insane, let's leave".
