Born in Sacramento, California, Caen worked for the San Francisco Chronicle from the late 1930s until his death, with an interruption from 1950 to 1958 during which he wrote for the San Francisco Examiner. His collection of essays entitled Baghdad-by-the-Bay was published in 1949.

Caen gained fame with his column "It's news to me", which was first published in 1938. His columns were known for their dry wit and his intimate knowledge of the goings-on in his city. Caen had a considerable influence on pop culture and its language; most notably, he is credited with inventing the term "beatnik" in a 1958 column and the word "hippie". He also playfully popularized many other less well known concepts and terms, such as Frisbeetarianism.

Caen often referred to his column as "three dot journalism" since it mostly consisted of short items broken up by ellipses. He ran the popular "Namephreaks" feature which presented people whose names were related to their occupations or hobbies (such as Nancy Canceller who worked the cancellation machines at the post office).

He regularly ran columns with this topic - one of his most popular. Entries were often submitted by one Strange DeJim, who frequently sent jokes to Caen and whom many suspected was not a real person but an alias Caen selected for himself. (Strange DeJim revealed himself after Caen's death as a writer who lives in San Francisco's Castro District.)

For many years, San Francisco had a double-decker freeway along much of its waterfront on the east side of the city, called the Embarcadero Freeway. Many residents considered to be an eyesore because it blocked views of the bay. Caen also loathed the freeway and he frequently lambasted it in his column (he called it the Dambarcadero). The 1989 Loma Prieta earthquake severely damaged the freeway and rather than repair it, it was decided to demolish it. In its place is a boulevard and Herb Caen Way, a wide promenade to the new stadium for his beloved San Francisco Giants.

Caen received a special award from the Pulitzer Prize board in 1996 "for his extraordinary and continuing contribution as a voice and conscience of his city". He appeared in a 1996 documentary called The Hippie Revolution. He died of lung cancer in San Francisco and his funeral was one of the most widely attended events in recent city history.