Hardly, as an adverb means "just", hardly, in this context is acting as a limiting modifier of the word ever. <em>Hardly ever</em> means very occasionally. We may use words like it to emphasize how infrequently an action takes place. You should note also that there could be other ways to give quite the same meaning, like when you use <em>rarely</em> or <em>seldom</em>, but, hardly ever has also a <u>negative</u> connotation.
