This should be an example of genetic drift.
The brown spotted deer would spread their genes more than the white spotted ones which would eventually lead to them completely overpowering genetically the white deer in future generations.
The factor that poses the greatest threat is loss of habitat. Have a nice day!