You were given the wrong answer. The correct answer is “des” since petits is in the plural form meaning their are many. So the sentence is “ce sont des petits cousins de mon père”
Un means my in French. Un is the masculine version of my. So, since you're talking about "my father's little cousins" you would have to go with a masculine version of un.
