Lucius Manlius Sargent (1786-1867) was the youngest child of Daniel Sargent and Mary Turner of Boston, Massachusetts. He married Mary Binney (d. 1824), daughter of Barnabas and Mary Binney of Philadelphia, in 1816. They had three children. He married (2) Sarah Cutler (d. 1868), daughter of Samuel and Sarah Dunn of Boston, in 1825. They had one son. The earliest known ancestor was William Sargent of Exeter, England, who was married to Mary Epes. He went from Exeter to Bridgetown, Barbadoes, and returned to England. His son William Sargent the 2nd came to Gloucester, Mass. previous to 1678, for in 1677 he married Mary (d. 1724), daughter of Peter Duncan.