Yes, the fundamental difference between these shades of quartz is in their mineralogy.
Pure quartz is made up of silica(SiO₂) in a framework silicate pattern. During the formation of these minerals, some other elements like certain metals can take up spaces in their lattices and crystal space. These other elements would serve as impurities in them and cause the different types of quartz we have. This is why some quartz appear rosy, some smoky, some milky e.t.c.
