Joseph Richardson was born in Rookby, Westmoreland, England 8 December 1816, one of the children of John and Barbara Richardson. He married Elizabeth Waters, daughter of William and Ann Watters 8 December 1840. On 28 March 1845 Joseph, Elizabeth, and their two children, John and Matthew, immigrated to New Orleans, arriving 12 May 1845. They settled in New Diggings, Wisconsin, where Ann, their third child, was born in 1846. They raised fourteen children.