"We are difficult. Human beings are difficult. We’re difficult to ourselves, we’re difficult to each other. And we are mysteries to ourselves, we are mysteries to each other. One encounters in any ordinary day far more real difficulty than one confronts in the most “intellectual” piece of work. Why is it believed that poetry, prose, painting, music should be less than we are? Why does music, why does poetry have to address us in simplified terms, when if such simplification were applied to a description of our own inner selves we would find it demeaning? I think art has a right—not an obligation—to be difficult if it wishes. And, since people generally go on from this to talk about elitism versus democracy, I would add that genuinely difficult art is truly democratic. And that tyranny requires simplification." Geoffrey Hill.
Generally don't love "challenging" music myself, but I'm glad it exists. There's a great line in a Rickie Lee Jones song somewhere - roughly (from memory) "It's not so much what you take from the painting as what you give yourself by what you leave", which hits it for me. The hard stuff asks us to participate. I'm not sure that it gets composers off the hook for writing a decent tune, but maybe that's a different argument.