They navigated by the sun, moon, stars, and landmarks.
This has happened more recently than you suppose. In the early Antarctic expeditions, which were in the 20th century, there were no maps ("Terra Incognita") and obviously compasses were useless as well.
As well, the South Pacific islanders, especially the Polynesians, always navigated by stars, ocean currents, and other natural signs. They were, and are, remarkable navigators.
