In this case, the function of their word is like a <u>possessive adjective</u>.
While it is not a pronoun, it is similar to a possessive pronoun.
The correct way to write the sentence would be: My dog chased <em>its</em> tail.
Unlike a pronoun, instead of replacing a noun, the possessive adjective <u>functions as an adjective</u>, and is placed before the noun it will modify.