Hamlet refuses to kill Claudius while praying, because he believes that Claudius's spirit will go to heaven since he is in the process of asking God for forgiveness. Hamlet knows that his father never had a chance to repent for his numerous sins before Claudius murdered him and does not think that sending Claudius's soul to heaven would be adequate revenge.
Explanation: Essentially, killing Claudius is not enough for Hamlet, who wishes to send Claudius's soul to hell for eternity. Hamlet hopes to kill his uncle while he is excessively drinking alcohol, gambling, or "in th' incestuous pleasure of his bed" in order to ensure that his soul will suffer in hell.
