Joseph E. Farrar was born to free parents in Charlottesville, Virginia in 1830. He worked as a carpenter under George W. Spooner, Proctor of the University of Virginia until 1858. Joseph married Rachel Willis Hill (d. 1908) of Charlottesville and moved to Richmond, Virginia. They had four children John E.C. Farrar, Daniel James Farrar, both contractors and builders, Maggie H.M. Farrar, the wife of Dr. S.H. Dismond and a teacher of music at the Hartshorn Memorial College, and Dr. Arthur W.G. Farrar, who studied medicine at Shaw University in Raleigh, NC. Joseph was deacon clerk and trustee of the Ebenezer Baptist Church in Richmond. He was elected to the Richmond City Council in 1885 as one of the first African Americans to hold a position with the Richmond City Government. He was treasurer and a director of the Friends Orphan Asylum, director of the Virginia Baptist Seminary, a founder and Director of the Colored Young Men's Christian Association. At one time, he was the largest black contractor of Richmond. He accumulated a good amount of real estate in Richmond.