The absolute worst for me was when I was just starting out. I would spend 12+ hours on the water and come home dead freaking tired and empty handed to always hear the neighbors ask "what'd ya catch?" "nothing" I'd say, and to that they would ask "why?"
Maybe it was just me being exhausted and all but that used to really piss me off.
