I think the Mitford family is fascinating. They were such a part of history. The Mitfords were an upper class family who seemed to be around every important person and event in England from the 1920s on.
I've read the non-fiction book The Sisters which first introduced me to the family. I also have read The Pursuit of Love and Love in a Cold Climate by Nancy Mitford. Those two books are on the 1001 Books You Must Read list and are stories based on the family. The Mitfords show up in other works of literature. Sir Oswald Mosley (sister Diana's husband) has been mentioned in the Maisie Dobbs series.
I really enjoyed the first part of Wait for Me! The book is the memoir of the youngest Mitford daughter, Deborah, who became Duchess of Devonshire. The first half is about Debo's childhood, teen, and early adult years and has lots of great stories about her family members.
However, when it got to the point where her father-in-law, the Duke of Devonshire, died and the family had to pay death taxes it went downhill quickly for me. They had to pay 80% taxes, which yes, is a huge portion of their wealth. But when she's saying how they had to sell off 42,000 acres here and another 11,000 there and all these fabulous works of art. Boo freaking hoo. She even had a catty remark about how a Rembrandt they had to sell was only a drawing/practice/etc and consequently not worth as much as previously thought and haha on the government for that. They still were beyond wealthy and didn't have to work. They even left England to live at their castle in Ireland because they didn't like the socialist/Labour governments of the time.
I had a really hard time caring about her after that point in the book and didn't finish it. I'm sure there were stories that were more interesting towards the end, but I just really didn't want to listen to the author anymore.
The Sisters by Mary Lovell is a great book about all the Mitford girls which you should definitely pick up. It's a great biography as well as covering so much history. I don't recommend Wait for Me! unless the money stuff won't bother you. Debo is not my favorite Mitford.