In 1935, Paul Mellon married Mary Conover Brown and the couple, who had two children, moved to Virginia. The Brick House at Oak Spring, Upperville, Virginia was completed in 1941 as their residence.

The architect, William Adams Delano of New York, had been persuaded by the Mellons to copy the principal exterior features of the Hammond-Harwood House in Annapolis, Maryland. The house was occupied by the family until the death of Mary Mellon in October of 1946. After his wife’s death Paul Mellon continued to live in this house.

On May 1,1948, he married Rachel Lambert Lloyd, known as “Bunny”, the former wife of Stacy Bancroft Lloyd. She is a descendant of the Lambert family who formulated and marketed Listerine and an heiress to the Warner-Lambert corporate fortune. They decided to turn the Brick House into a library and gallery.