He was married twice, first to Francis Mary Gascoyne on 2 February 1821 and later to Mary Catherine Sackville-West, daughter of the 5th Earl de la Warr and Lady Elizabeth Sackville, Baroness Buckhurst, on 29 April 1847. He had six children with his first wife and five with his second. His third son, Robert, who succeeded him as third marquess, was Prime Minister of the United Kingdom three times between 1885 and 1902.
