After all, on one end of the scale, poetry has gone from being something that you did in order to Write Your Name Large Across the Sky to a carefully gated medium that requires years of apprenticeship to produce meticulous golden lines that up to 10 people will ever voluntarily read. The last time I stumbled upon a poetry reading, the attendees were students of the poet who were there in the hopes of extra credit.
Me, whom no Muse of heavenly birth inspires,
No judgment tempers when rash genius fires;
Who boast no merit but mere knack of rhyme,
Short gleams of sense, and satire out of time;
Who cannot follow where trim fancy leads,
By prattling streams, o’er flower-empurpled meads;
Who often, but without success, have pray’d
For apt Alliteration’s artful aid;
Who would, but cannot, with a master’s skill,
Coin fine new epithets, which mean no ill:
Me, thus uncouth, thus every way unfit
For pacing poesy, and ambling wit,
Taste with contempt beholds, nor deigns to place
Amongst the lowest of her favour’d race.
Emerson’s remark helps suggest the all-important difference: “It is not meters, but a meter-making argument that makes a poem.” In this kind of poem the poet establishes regularity only to depart from it expressively. When he does compose a metrically regular line it is not because the metrical scheme tells him to, but because something in the matter he is embodying impels him toward a momentary regularity. […]
The poet working in free verse has already chosen to eschew one of the the most basic expressive techniques in poetry.
Paul Fussell: Poetic Meter and Poetic Form
(from Chapter 3, Metrical Variations)
Because the combination of song writer and recorded music displaced it. A song is much more accessible to the average person than a poem, and technology made songs widely available. Poetry responded to its diminished status by retreating into narcissistic incoherence. Sure the philistines may not recognize the true greatness of modern poetry, but at least poets could take comfort in their own self-declared cultural superiority. When public rejection became a necessary characteristic of great poetry, then there was no longer any hope.
It’s not correct to say a poet is the same as a song writer. It’s easier to write lyrics for a song because the song writer can lean upon the music. The poet has only words (and at one time such archaic concepts as rhyme and rhythm and meaning and significant subject matter) to carry his message. Poetry is a much harder art form. But better Dylan than a modern poet who composes what passes for poetry in this day and age. Better to let the dead decompose undisturbed.