White was born in Homer, New York and educated at Yale University. At Yale, he was a classmate of Daniel Coit Gilman, who would later serve as first president of Johns Hopkins University. The two were members of the Skull and Bones secret society, and would remain close friends. He was also a member of the Alpha Sigma Phi fraternity, serving as editor of the fraternity publication, The Tomahawk.

After graduating from Yale in 1853, White spent three years studying in Europe before returning to the United States as a professor of history and English literature at the University of Michigan.

In 1865, White and Western Union tycoon Ezra Cornell founded Cornell University on Cornell's estate in Ithaca, New York. White became the school's first president, and his farsighted leadership set the university on the path to becoming an elite educational institution, with particular excellence in agricultural research and engineering.

After 14 years at Cornell, White resigned to serve as the U.S. Minister to first Germany (1879-1881) and later Russia (1892-1894), and as the first U.S. Ambassador to Germany (1897-1902).

While serving in Russia, White—a noted bibliophile—made the acquaintance of author Leo Tolstoy. Tolstoy's fascination with Mormonism sparked a similar interest in White, who, like most educated Americans at the time, had previously regarded the Latter-Day Saints (LDS) as a dangerous, deviant cult. Upon his return to the United States, White took advantage of Cornell's proximity to the original Mormon heartland near Rochester to amass a collection of LDS memorabilia (including many original copies of the Book of Mormon) unmatched by any other institution save the church itself and its university, Brigham Young University.

In 1891, Leland and Jane Stanford asked White to serve as the first president of the university they had founded in Palo Alto, CA. Although he refused their offer, he did recommend his former student David Starr Jordan.

White died in Ithaca and was interred in Sage Chapel at Cornell.

Contribution to the conflict thesis
At the time of Cornell's founding, White announced that it would be "an asylum for Science—where truth shall be sought for truth's sake, not stretched or cut exactly to fit Revealed Religion" (Lindberg and Numbers 1986, pp. 2-3). Up to that time, American universities were exclusively religious institutions, and generally focused on the liberal arts and religious training (though they were not explicitly antagonistic to science). In 1869 White gave a lecture on "The Battle-Fields of Science", arguing that history showed the negative outcomes resulting from any attempt on the part of religion to interfere with the progress of science.

Over the next 30 years he refined his analysis, expanding his case studies to include nearly every field of science over the entire history of Christianity, but also narrowing his target from "religion" through "ecclesiasticism" to "dogmatic theology." The final result was the two-volume History of the Warfare of Science with Theology in Christendom (1896). Initially less popular than John William Draper's History of the Conflict between Religion and Science (1874), White's book became an extremely influential text on the relationship between religion and science.

The premise of the book—known as the conflict thesis—was very prevalent among historains through the 1960s. Since the 70s and 80s, many historians of science have reevaluated the history of science and religion, finding little evidence for White's claims of widespread conflict; instead, they often blame White for perpetuating a number of scientific myths, such as the idea that Christopher Columbus had to overcome widespread belief in a flat earth.