Realising the strategic value of the site of the old Roman fort a new castle was raised within its remains by the Normans.
The Norman Lord of Gloucester Robert Fitzhamon, drawn into the quarrels of the Welsh princes, had defeated Rhys ap Tewdur in 1093, leaving him as Lord of Glamorgan. The Normans concentrated their defensive works into the western half of the site, which became the 'inner' ward. At the northern end of this part, Fitzhamon built a 'motte', or mound, 40 feet high. This Keep was probably surmounted by a timber stockade giving shelter and protection to the wooden buildings which housed the lord, his household and his garrison.
Robert Fitzhamon died of wounds received in battle in 1107, and his daughter and heiress, Mabel, married Robert, the natural son of King Henry I of England. King Henry raised Robert to the Earldom of Gloucester, and made him Lord of Glamorgan, in 1122. The Earl of Gloucester was lauded on all sides as a brave soldier, wise statesman and patron of the arts, and is credited with having built the first stone keep of Cardiff Castle. At King Henry's request, he imprisoned another Robert in this keep - the second Duke of Normandy and his father's older brother - from 1126 until Robert of Normandy's death in 1134.
Robert, sometimes known as 'The Consul' died in 1147 and was succeeded by his son William. When William died without male heir in 1183, the lordship passed to Prince (later King) John through his betrothal to William’s daughter, Isabel. John divorced Isabel but retained the lordship until her second marriage to the Earl of Essex. In 1217 the lordship passed to Gilbert de Clare, son of Isabel's sister, Amicia, and the descendant of a noble family which claimed kinship with William the Conqueror.