Colbert took over from David Letterman in 2015 but he leaves amid political controversy with parent company Paramount.