Because:
Not everything around you has the correct moisture to grow mold. Places like a shower, wood, and even toilets do, since water will soak into them, or the water is trapped. If you don't clean your shower and toilet, you will notice mold. If you look at a fence, there is probably hints of mold on it.
Something like a shelf cannot grow mold, since it usually has paint on it, and the moisture won't go through something like paint.
Another possible answer: Since the areas around you don't always have the correct moisture, mold will not grow on everything. It can grow on some stuff in your house, but not everything around you.
If you need any more help, let me know !
