The palace is located next to the Church of St John the Baptist and the former Vilnius Academy, in the neighbourhood of the defunct Radziwill residence called Kardynalia, as well as the former Lacki and Sapieha houses and the Brzostowski palace. Swietojanska Street connected with Zamkowa Street and the representative via regia route, facing south to the market square with the town hall and St Casimir's Church, and north to the castle and cathedral. Slightly further on, between the Ostrą Brama (Gates of Dawn) and the Rudnicka Gate, there was a quarter connected with the Carmelites, where Stefan Pac (1587-1640), together with his wife Anna Maria Ancilla of Rudomin Dusiacki (died 1643), founded two churches - St Teresa's and St Joseph's (demolished after the January Uprising). It was this enterprising Lithuanian treasurer, later Grand Chancellor of Lithuania, who elevated his family to the elite of the Rzeczpospolita, who bought the estate in 1628. From then until 1783, it remained the property of the Pacas family, and then, for a short period of less than 20 years, of the Sapieha family.

Stefan Pac renovated the building after buying it. The work was completed before 1633, for from that year onwards he willingly and, as we read in the sources, "generously" received guests at his palace in Vilnius. The receptions began with the first wedding of his daughter Sophia on 24 July 1633, at which 'the whole court was present at the ceremony and feast, dedicating the day to merriment, dancing and Bacchus'. Her wedding to the Lithuanian equerry Jan Kazimierz Chodkiewicz 'with great pomp' took place three years later, and was graced by Anna Catherine Vasa. "The tightness of the space could not accommodate this crowd. At six o'clock in the evening, amidst confusion and quarrels, and even wounds to some, the royal returned to the castle, and the guests thus ended the day and the month". Elsewhere, the indispensable diarist Albrycht Stanislaw Radziwill noted: "the Lithuanian sub-chancellor received us all for almost the whole day with a splendid feast; wine, music, the eagerness of the revelers; he did not let the well-fed and watered guests go until sunset".

The Muscovite occupation (1655-1660) contributed to the destruction of the palace, and the residence had already been rebuilt by Stefan Pac's younger son, Lithuanian Chancellor Christopher Sigismund (1621-1684). Aiming to give the capital's seat an appropriate representation, he could not extend the front building due to the nearby tenement houses, so he built two-storey side wings (he extended the eastern one) enclosing an irregular trapezoidal courtyard. Work was carried out from the early 1760s by, among others, a mason who was called a German or "Olender" who also did "stucco work". A carpenter imported from Königsberg also worked on the palace.

Profound changes in the appearance of the Vilnius palace took place after 1748, when it burned down in the great fire of the city. At the time, it belonged to Antoni Michał (1715/16-1774) and Teresa Barbara of Radziwiłł (1714-1780), a wealthy widow married to Pac in 1745. The work undertaken by the couple made the palace one of the city's chicest, resembling the then fashionable French hôtels. From then on, it was also teeming with a rich social life, and the Pacs willingly and frequently held various events there.

During the works undertaken, the parade courtyard was closed off with a brick, two-storey outbuilding with a mezzanine, single-arched with a gate on the axis. As it also served a representative and residential function, its facades were framed with Tuscan pilasters in great order. The first floor housed rooms in an amphitheatre, including the boudoirs so fashionable at the time.

The front building was also rebuilt and given a mezzanine. A "choir" for the band was organised in the grand reception hall, located on the first floor on the façade side. The entrance façade received a new Rococo decoration with Ionic pilasters, a stucco decoration in the form of a panoply with a coat-of-arms cartouche in the tympanum, and a gate framed by a volute decoration. The ornaments were intended to recall the soldierly deeds of Antoni Michal Pac's ancestors and probably to create his own knightly dignity, which he did not seem to possess: although he was a colonel in the Petrograd and Hussar flags of the Great Lithuanian Bulla, he became famous for escaping through the window during the Sejm tumult.

The palace became the property of the Vilnius magistrate in 1801, and in 1816 or 1831 the building was taken over by the Tsarist state to accommodate gubernial offices. Before World War I, the palace housed the Russkij Mir restaurant, as well as shops and warehouses. In 1918, the Lithuanian association "Rūta" operated in the building, and in 1923, the bookshop of the "Švyturys" company. In 1944, as a result of warfare, the palace was destroyed and its immediate surroundings were transformed. In 2007, the former seat of the Pacs was purchased by the Republic of Poland, arranging an embassy in the building, carefully conserving the palace in 2013-2017.