The song, as Cohen wrote it, is rich with references to Jewish Scriptures, including further references to former shepherd and now King David and his stolen love Bathsheba (2 Samuel 11 v 2). David's fall from God's favour to his eventual restoration to God as one of Israel's most famous kings.
A stalk or bunch of celery refers to a number of branches or ribs with a central heart. Americans use the word stalk to refer to a single branch or rib, when the word stalk really means a bunch of celery.