The skepticism about the empire of Ghana and the accounts for it is nothing weird because the majority of what is written about it is from two people from the same place, that had totally different views and interpretations on the things, and came from different culture.
Very often in the historical text, the people that wrote something have been very subjective, not objective. Thus the writings of these two Arab geographers can be very misleading, as they described what they saw with their own eyes, but also with using their own perception. That has proven numerous times to give very inaccurate depictions of a society and culture, like the depictions of the Romans for the Celts, or of the Greeks for the Scythian female warriors that they named Amazons.
There's only one point of view unfortunately, and it is always much more reliable when multiple writings are available from people from multiple different backgrounds, or the best scenario if it is writings from the people in question.
