In about 1300 Sir John de Broughton built his manor house in a sheltered site at the junction of three streams and surrounded it with a substantial moat.
William of Wykeham, Bishop of Winchester and Chancellor of England bought the house in 1377. It then passed to William’s great-nephew Sir Thomas Wykeham and thence to Sir Thomas’s granddaughter, Margaret, who married Sir William Fiennes, later the 2nd Lord Saye & Sele, in 1448.
Sir Thomas Wykeham obtained a licence to ‘crenellate and embattle’ in 1406: he added the battlemented wall to the gatehouse, thus giving the medieval house a military appearance – these changes allowed the manor house to be called a castle.
In 1554 Richard Fiennes completed a major reconstruction. He raised the roof to accommodate two floors above the Great Hall, building two staircase projections to the south and adding – on the foundations of the medieval kitchens – two rooms which form the west wing. After his death in 1573 his son, Richard, continued the embellishment of the interior, recording the date 1599 on the plaster ceiling in the Great Parlour.
The next period of building work came as a result of Civil War damage. After the nearby Battle of Edgehill in 1642 the local superiority of the Royalists enabled them to lay siege to the Castle which was captured and occupied. The need for repairs is reflected in the date 1655 on the gatehouse. Further outbuildings may have been damaged or destroyed and the Castle may have remained in poor condition: in the late 1690’s Celia Fiennes describes “my brother Saye’s house being much left to decay and ruine”.
The 18th century was by contrast uneventful; but in the 19th century, William Thomas, 15th Lord Saye & Sele, indulged in a life of frivolity and extravagance as one of the set surrounding the Prince Regent and the Count d’Orsay. The family then lived at the more fashionable Belvedere at Erith in Kent and their neglect of the Castle caused it to be noted in 1819 that the rooms were ‘daily dilapidating from misuse’. In 1837 the bulk of the contents were disposed of in a twelve-day sale, the last item being the swans on the moat.
It is ironic that the squandering of the family fortune in the Regency period almost certainly saved Broughton from the architectural excesses of the Victorian age. William Thomas’s successor, Frederick, 16th Lord Saye & Sele, carried out vital repair work in the 1860’s with the architect George Gilbert Scott. Unfortunately, further neglect followed when John, 17th Lord Saye & Sele directed his available funds at racehorses rather than the Castle. He let the Castle in 1886 to the Gordon-Lennox family and they invested in many enhancements to the gardens.
When the Gordon-Lennox’s lease expired in 1912 the Fiennes family returned. There remained a shortage of resources for the repair and maintenance of the Castle, but the second half of the last century was characterised by major restoration. In 1956 financial assistance received through the Historic Buildings Council enabled the stone-tiled roof to be renewed. In the eleven years between 1983 and 1994, in a programme led by Nathaniel, 21st Lord Saye & Sele, continuous stonework and other restoration took place towards which English Heritage gave generous support.