IF you are dilating about the origin, which you didn't exactly state, the new coordinates are A' (1/5, 1), B' (2/5, 1/5), C' (0, 4/5). Dilation is all about distance, so you have to move the new coordinates to whatever the scale factor is times the x and y distance. For example, A is at (1, 5), so 1/5 of 1 is 1/5, and 1/5 of 5 is 1, hence your new coordinate for A. It is a fifth of the distance from the origin as its preimage is.
