Under the Restoration, in 1817, some repair work was begun. The following year the Pavillon de Breteuil was the subject of various proposals: the Comte de Pradel, Director of the Royal Household, offered it to the Comte de Hoguerté; the Marquis de Vernon, first equerry of the King, requested it for an equerry of the Comte d'Artois. Despite the repairs of the preceding year, the Pavillon de Breteuil was not yet habitable. Major work and heavy expenditure were required and nothing was left of the furnishings. It was even suggested that the only way to make use of the Pavillon de Breteuil would be to make it into lodgings for grooms!

In 1820 there was some question of putting the Pavillon at the disposal of the Duchesse de Berry but she insisted on complete redecoration. The cost would have been 7000 francs; this was thought to be too expensive and the idea was dropped. The 'Garde des Sceaux', de Serre, finding his lodgings at the Château de Saint-Cloud inadequate when the Court was in residence, expressed a desire to live in the Pavillon de Breteuil and requested that it be repaired. He was less demanding, doing only the essential repairs and putting up ordinary wall paper to avoid re-doing the paintwork; the total expenditure came to about 1500 francs.

In 1822 the Pavillon de Breteuil passed to the Governor of the Château de Saint-Cloud, the Vicomte d'Agoult. A one-time émigré who was in the Prince's army during the campaign of 1792, the Vicomte accompanied Louis XVIII to Verona, to Mittau and then to England. Promoted Lieutenant-Général on the return of the Bourbons, he became the first equerry to the Duchesse d'Angoulême and then in 1821 was made Governor of Saint-Cloud. For his arrival the Pavillon de Breteuil was once again refurbished. The same year the Princesse Esterházy, wife of the Austrian Ambassador in Paris, spent several short stays in the Pavillon de Breteuil. The Duchesse d'Angoulême, living in the Château de Saint-Cloud, visited her frequently on these occasions.

During the year 1823-1824 the Duc de Blacas d'Aulps, first gentleman of the King's Bed'Chamber, lived at the Pavillon de Breteuil with his family. Also an émigré, the Duc, at the Restoration, was made Minister of the King's Household, Secretary of State, Grand Master of the Robes and Intendant General of the Crown buildings. As Minister of the King's Household he had charge of the refurbishment of the Château de Saint-Cloud for the first visit of Louis XVIII in 1816.

The Pavillon de Breteuil was inhabited for a few months in 1830-1831 by Maréchal, provisional Governor of the Château. On his departure the building was left unoccupied but for a caretaker who looked after the remaining valuable furniture. In 1831 the Pavillon de Breteuil was put at the disposal of the Duc de Castries, Governor of the Château de Meudon, who had been obliged to give up his usual lodgings at Meudon in favour of Dom Pedro, Emperor of Brazil, during the latter's stay in France. The next record of the Pavillon de Breteuil shows that in 1842 it was given to the Comte de Montalivet who, having left the Ministry of the Interior in 1839, was appointed 'Intendant' of the Civil List, a post he was to continue to occupy until 1848. In November 1842 a note in the Journal des débats mentions that the Comte de Montalivet was laid up at the Pavillon de Breteuil with an attack of gout!