Not to excuse Josh Duggar, but if he looked like, say, a youthful Warren Beatty rather than a schlub, he'd come in for very different treatment. If you're plain and plump, and you're caught in sex scandal, a lot of people hold you in contempt. If, however, you're a handsome playboy, many of the same people may even have a sneaking admiration. Image literally has everything to do with how some people evaluate misconduct.
Or take people like Bill Clinton and Donald Trump. They aren't even handsome. But they project power. That, combined with their bad boy image and swinger lifestyle makes them objects of vicarious admiration.