A theory posted on Reddit suggests Dementors were drawn to Harry because they could sense the piece of Voldemort’s soul living in him. As such, Dementors could have affected him more because there was literally an extra piece of (someone else’s) soul in him, and one that was loaded with bad memories.
He fears them more because he has to relive the memory of his parents’ deaths each time he encounters them. We have nothing to fear but fear itself. He’s not frightened of the dementors, he’s frightened of what they inflict on him.