The Palace is located at the intersection of three streets: Dominikonų g., Šv. Jono g., Universiteto g. (Dominican, St. John's and Bishop's Streets), next to the former Papal Alumni for Muscovites and Ruthenians, at the back of the former Bishop's Palace (Governor's Palace, now the President's Palace), in the vicinity of St. John the Baptist Church and Vilnius Academy, and the churches of the Holy Trinity and the Holy Spirit. The plots of land together with the buildings, ruined after the Muscovite occupation (1655-1660), were acquired in 1660. The land, together with his wife Rachela, née Dunin Rajeckis, was purchased by the registrar of the Grand Duchy of Lithuania and founder of the Church of Canons Regular of Penance in Michališkės, royal envoy and soon afterwards castellan (1681) and voivode of Trakai (1684) Cyprian Pavel Bžostovský (1612-1688), the founder of the family power of the Strzemie coat of arms, a man of, as it was written at the time, "great wit, prudence and deep caution".

The seat was inherited by his son Jan Władysław (1646-1710), Castellan of Troki, who renovated the building after the fire, and the work was continued by the Lithuanian scribe Józef (d. 1745), of "bloodthirsty temperament". The seat burned down again in 1748 and 1760, when it was already the property of the Livonian voivode Stanislav Brzostowski (1733-1769). He ceded the real estate to his brother, Pavel Ksawery, canon of Vilnius, writer and future referendary of Lithuania (1774). A publicist and translator, as well as a generous protector of the literati, he was the founder of the Peasant State (Pavlovian Republic, 1769). As late as 1762, he took charge of the renovation of the palace, which was carried out for him by Carlo Spampani. The decoration of the ground-floor room - a study or bedroom - dates from this year, with rocaille, Strzemię coats of arms and a cartouche with an inscription indicating the owner of the residence painted on the walls and ceiling. The Rococo decoration of the hall suggests that the other rooms received such a character. This is because Brzostowski undertook work to completely modernise the interiors and give a new look to the facades, as well as to brick outbuildings. At the end of these works, in 1769, he founded a plaque in memory not only of himself, but also of the Brzostowski family - builders and owners of the Vilnius palace. It is located above the side door on the side of the courtyard, which once led to the chapel on the ground floor of the palace. The location of the plaque, today completely incomprehensible due to the different function of the rooms, was therefore not accidental.

As mentioned, the Vilnius residence acquired its basic shape in the 1760s as a two-storey building, situated on an east-west axis in a frontage development. The basement body took on an irregular rectangular shape with sections of side wings facing the courtyard. The body was planned as a two-aisle structure with vaults in both its storeys, the ones on the first floor were probably established during the reconstruction undertaken by Paweł Ksawery Brzostowski. From the courtyard, two-storey and three-axial arcade-pillar loggias with Tuscan pilasters (glazed and partly walled) were organised in the part of the gate passage.

Despite the successful reconstruction, Paweł Ksawery Brzostowski sold the palace in 1772 to the chamberlain of Samogitia, Jakub Ignacy Nagórski (1734-1799), a knight of the orders of St. Stanislaus and the White Eagle associated with Stanisław August. He introduced his family into the elite of the Grand Duchy of Lithuania, proudly titling himself "Count of Great and Small Corcians". Hence, in the gate portal the coat of arms of the Brzostowskis was replaced at that time with that of the Pobóg Nagorski family (erroneously reconstructed). It was then that one of the ground floor rooms was decorated with a paludamentum arbor type painting. Such decoration was typical mainly for residences connected with garden layouts. The Vilnius realisation should therefore be considered original, as it was realised in the city.

In 1802, the palace was sold to the Oginskis. It was inhabited by Maria de Nèri (1788-1851), widow of Kajetan Nagórski (after 1760 - 1802) and second wife of Michał Kleofas Oginski (1765-1833). The (partially) preserved interior decoration of the first floor, including the illusionist columns against the rusticated wall, dates from this period. In the years 1888-1890 the palace was renovated by the St Petersburg-educated architect Cyprian Maculewicz (1830 - before 1906?), who probably built annexes and perhaps also enlarged them, eventually closing the courtyard.

The Oginskis rented some of the rooms. From 1887, a six-room flat was occupied by Władysław Zahorski (1858-1927), a physician, an expert on the history of Vilnius and its monuments, as well as an organiser of the Society of Friends of Science (1906). Today, the palace houses flats, offices and shops, as well as a pharmacy. Some of the outbuildings remain abandoned.