He had been styled Earl of March, before inheriting the title of Duke of Richmond from his father in 1860. He was a short-lived cricket player for Oxford University, presumably his alma mater. He married Frances Harriett Greville on November 28, 1843, and they had one child, Charles Henry, who would succeed his father as 7th Duke of Richmond.