Emerson’s remark helps suggest the all-important difference: “It is not meters, but a meter-making argument that makes a poem.” In this kind of poem the poet establishes regularity only to depart from it expressively. When he does compose a metrically regular line it is not because the metrical scheme tells him to, but because something in the matter he is embodying impels him toward a momentary regularity. […]
The poet working in free verse has already chosen to eschew one of the the most basic expressive techniques in poetry.
Paul Fussell: Poetic Meter and Poetic Form
(from Chapter 3, Metrical Variations)
Because the combination of song writer and recorded music displaced it. A song is much more accessible to the average person than a poem, and technology made songs widely available. Poetry responded to its diminished status by retreating into narcissistic incoherence. Sure the philistines may not recognize the true greatness of modern poetry, but at least poets could take comfort in their own self-declared cultural superiority. When public rejection became a necessary characteristic of great poetry, then there was no longer any hope.
It’s not correct to say a poet is the same as a song writer. It’s easier to write lyrics for a song because the song writer can lean upon the music. The poet has only words (and at one time such archaic concepts as rhyme and rhythm and meaning and significant subject matter) to carry his message. Poetry is a much harder art form. But better Dylan than a modern poet who composes what passes for poetry in this day and age. Better to let the dead decompose undisturbed.
The current culture of instant memes, meaning and 30-second bites of information is forcing readers to abandon anything that involves several reads. Poetry needs to simmer, and today’s society only accommodates things that can be thrown into the microwave for a few seconds and then consumed on the drive to work. We have mobile phones, social networking sites and a lot of other distractions to fill our time. All that is left of poetry in the mainstream media is pop music, which can be overly stylized. Poetry has a meditative aspect to it. It’s about taking a poem and going somewhere quiet to mull it over. People are no longer interested in that, their attention spans crave constant stimulation.