The palace organised by Michał Kazimierz Pac, Grand Hetman of Lithuania and Voivode of Vilnius, one of the most attractive in Vilnius, was the heritage of the Pac family until 1831. The palace was located in a prestigious part of the city, right in its centre. Nearby were the palaces of the Radziwiłł and Chodkiewicz families, and opposite them stood the Church of St Nicholas. Slightly further away was St Casimir's Church and the Town Hall Square.

The beginnings of the brick buildings on the site where the Pacs' palace was built on two plots of land date back to the turn of the 15th and 16th centuries. The old walls were used during the construction of the new residence by Giovanni Battista Frediani, who, as a military engineer, served under the command of Michał Kazimierz Pacs. He was supported by craftsmen from Vilnius, but also from Krakow, brought by Pac to build the church of St Peter and St Paul in Antokol.

The mere height of the residence, rising to three storeys, was impressive, but so was the architectural programme with its clearly visible representative function. The palace was set up as a frontage building, on an east-west axis, with two annexes to the courtyard: a square and a rectangular one. The southern one contained, among other things, an additional staircase, while the northern one had a kitchen downstairs and a cupboard on the first floor. An arcaded arcade (walled up) was spanned at the rear elevation in the ground floor.

The parade gate was situated on the axis of the main front elevation. It was framed by pairs of Tuscan columns that supported a shallow stone balcony at the most important floor hall (destroyed at the turn of the 20th century). This would therefore be one of the earliest, and perhaps the first, examples in the Republic of the combination of a gate/portal and the windows of a representative room with a balcony, a solution that would become typical of the 18th century. The windows of the facade, more than 31 m wide, were framed in the second storey by raked frames with pediments, of which the central one, marking the axis of symmetry and the main hall of the palace, was designed as segmental. Above it, until the early 1830s, was the Pacs' coat of arms, Gozdawa, made of black Malopolska stone.

While the two-bay, vaulted ground floor was intended from the very beginning as a rental for various storerooms and stalls, the representative rooms were organised on the ground floor. The parade character of the first floor was also marked in the rear façade by a more elaborate window design.

The façades did not undergo any changes in the following centuries. However, the interiors were modernised by adapting them to fashions and introducing various amenities. The last and most important transformation of the palace interiors took place during the reign of Ludwik Michał Pac (1780-1835), a participant in the Napoleonic campaign, one of the commanders of the November Uprising, a man committed to Polish affairs and a connoisseur of culture and an accomplished collector of old art.

In the years 1810-1811, he remodelled the interiors in the spirit of classicism and richly furnished the palace, creating an exclusive urban residence. Particular attention was paid to the first floor and the staircase leading up to it. In the "blue boudoir", Ionic columns with marbled shafts were molded. An 'orangery' study was organised next to the great hall, with a billiard room next to it. The rooms of the rear tract were converted into an anteroom and a dining room. The last was converted into a dressing room connected to a bedroom adjacent to the playroom.

The rooms acquired landscape (Jan Rustem with his pupils, staircase), illusionist (including rustication in the staircase, columns with beams in the billiard room and in the playroom) and mythological (playroom, dining room, billiard room, dressing room) paintings. The representative staircase was decorated with a niche in which a plaster statue of Minerva was placed, made in the workshop of the former artist of King Stanisław August Poniatowski - André Le Brun, who ran a sculpture school at the then Imperial Vilnius University in 1803-1811.

New floors and cookers were also laid, as well as glazed windows and new entrances to the staircases on the courtyard side. The work was carried out by, among others, Vilnius craftsmen, including Jurewicz, Witkowski and Zilinski, and was coordinated by Giovanni Boretti of Lombardy, a master mason, a noted contractor for numerous repairs to church and secular buildings, to whom the marbled columns of the boudoir should be associated, among others.

After the confiscation of Ludwik Michał Pac's property, the building housed the district treasury and division headquarters for a while, and then the post office. After the Second World War, the former Pacs' residence served various functions, and since 2018 it has housed an exclusive hotel. The latter function is an apt reference to the prestige that the Pacs' headquarters enjoyed throughout its history. It hosted the most eminent figures visiting Vilnius, including John III Sobieski (1688), Tsar Peter I (1705) and Nuncio Saluzza de Corigliano (1787). Tsar Alexander I and Napoleon attended the receptions at Ludwik Michał Pac's house. The latter turned up at around 9 p.m. at a grand ball organised by Pac on 14 July 1812 to celebrate the incorporation of Lithuania into the Duchy of Warsaw. He only stayed for an hour, while the rest of the revelers left at around 5am. King Joachim Murat of Naples lived in the palace in the same year, as did Antoine Henri de Jomini, commandant of Vilnius and governor of Smolensk, who oversaw the construction of Tsarist fortifications in the suburb of Snipištia. In the years 1818-1822, the palace annexe occupied by Onufry Pietraszkiewicz was the venue for meetings of the Philomaths and Philarets, attended by Adam Mickiewicz and Tomasz Zan.