The King’s brother is said to be in talks about leaving Royal Lodge following the furore over the ‘peppercorn’ rent for the 30-room mansion.