In many ways, the new lords wrote a sordid chapter in the history of the castle.
On the death of the young Gilbert de Clare in 1314, Glamorgan and Cardiff passed for a while into royal custody, and during the period when Pain de Tuberville, Lord of Coity, served as 'custos' all Welshmen were removed from office. These included Llewelyn Bren, a great-grandson of the Welsh hero Ifor Bach and a good friend of the last young lord.
Edward II's estranged wife Isabella (the 'She-wolf of France') returned from France in 1326 and directed her army against her husband's favourites. Edward and Hugh Despenser fled for refuge to Glamorgan but were captured some 10 miles to the west of the castle. Edward was eventually taken to Berkley Castle in Gloucestershire and was foully murdered in 1327. Retribution against the Despensers came more swiftly; Hugh was hanged at Hereford in 1326, one of the charges against him was that of the murder of Llewelyn Bren. On his widow’s death in 1337 their son, Hugh Despenser II succeeded to the lordship. This Hugh's death in 1349 saw his nephew Edward come into possession of Cardiff Castle. Edward regained some family prestige through his career as a distinguished soldier. However, the last Despenser lord was Thomas, who like his ancestors, met a violent death, being executed at Bristol in 1400.
The Welsh rebellion of Owain Glyndwr broke out in the year 1400, and 4 years later Owain broke through the west gate of Cardiff and set fire to the castle and town. The ferocity of the attack was in some measure due to the hatred still felt by the Welsh against the Despensers for the murder of Llewelyn Bren in 1317. The only place spared was the monastery of the Grey Friars, whose monks, it was said, had given a Christian burial to Llewelyn's body.
The heiress of the castle, Isabel Despenser, daughter of Thomas, had married Richard Beauchamp, Earl of Worcester, and thus began a new chapter in the story of the castle.