Question

What did I do wrong in the highlighted parts

Answer 1

You did
D=(x2-x1)²+(y2-y1)²
it should be
D=$\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}$

and BC, you did (-4)²=-16, it should be equal to 4

if you do the math which is evaluate

A=(5,3)
B=(1,5)
C=(3,0)

AB=$\sqrt{(1-5)^2+(5-3)^2}= 2\sqrt{5}$
BC=$\sqrt{(3-1)^2+(0-5)^2}= \sqrt{29}$
CA=$\sqrt{(5-3)^2+(3-0)^2}= \sqrt{13}$

add them to get total