Sometimes, the judgement of the mind, which keeps track of what is good and bad to do, overdoes it and forgets to take your feelings into consideration. Might be a good idea to tell this guy to back off at such times.