By "Cain's heresay" he was referring to the famous phrase "I am not my brother's keeper." Cain used it meaning that he was not responsible for what his brother did, and if his brother made bad decisions the consequences should fall on him, and not Cain, that t wasn't his job to babysit his brother. By alluding to this quote in the passage Mr. Utterson was saying that it is not his job to monitor or judge anyone, that he is not anyone's keeper or babysitter. it is not his job to judge the actions of others as right or wrong, and he "let's his brother go to the devil in his own way." Meaning if you want to make bad choices he is not responsible for what you do, and will not judge you for doing it. he's basically saying we're all grown ups here, do what you want to do, and I'll be your friend either way....we are all responsible for our own choices good or bad.
