In aristocratic ages each people as well as each individual is prone to stand separate and aloof from all others. In democratic ages the extreme fluctuations of men and the impatience of their desires keep them perpetually on the move, so that the inhabitants of different countries intermingle, see, listen to, and borrow from each other. It is not only the members of the same community then, who grow more alike; communities themselves are assimilated to one another, and the whole assemblage presents to the eye of the spectator one vast democracy, each citizen of which is a nation. This displays the aspect of mankind for the first time in the broadest light. All that belongs to the existence of the human race taken as a whole, to its vicissitudes and its future, becomes an abundant mine of poetry.
Alexis De Tocqueville: Democracy in America, published 1835–1840
Emerson’s remark helps suggest the all-important difference: “It is not meters, but a meter-making argument that makes a poem.” In this kind of poem the poet establishes regularity only to depart from it expressively. When he does compose a metrically regular line it is not because the metrical scheme tells him to, but because something in the matter he is embodying impels him toward a momentary regularity. […]
The poet working in free verse has already chosen to eschew one of the the most basic expressive techniques in poetry.
Paul Fussell: Poetic Meter and Poetic Form
(from Chapter 3, Metrical Variations)
Because the combination of song writer and recorded music displaced it. A song is much more accessible to the average person than a poem, and technology made songs widely available. Poetry responded to its diminished status by retreating into narcissistic incoherence. Sure the philistines may not recognize the true greatness of modern poetry, but at least poets could take comfort in their own self-declared cultural superiority. When public rejection became a necessary characteristic of great poetry, then there was no longer any hope.
It’s not correct to say a poet is the same as a song writer. It’s easier to write lyrics for a song because the song writer can lean upon the music. The poet has only words (and at one time such archaic concepts as rhyme and rhythm and meaning and significant subject matter) to carry his message. Poetry is a much harder art form. But better Dylan than a modern poet who composes what passes for poetry in this day and age. Better to let the dead decompose undisturbed.
The current culture of instant memes, meaning and 30-second bites of information is forcing readers to abandon anything that involves several reads. Poetry needs to simmer, and today’s society only accommodates things that can be thrown into the microwave for a few seconds and then consumed on the drive to work. We have mobile phones, social networking sites and a lot of other distractions to fill our time. All that is left of poetry in the mainstream media is pop music, which can be overly stylized. Poetry has a meditative aspect to it. It’s about taking a poem and going somewhere quiet to mull it over. People are no longer interested in that, their attention spans crave constant stimulation.