In 1496, Lithuanian Grand Duke Alexander granted Druja to Timothy Massalsky, who built a castle there or, what we do not know for sure, extended the earlier fortifications existing there. In 1611, fragments of the village with the castle were acquired by the Grand Chancellor of Lithuania Lev Sapieha (1557-1633), and for almost two centuries the estate remained in the family's hands. The same chancellor built the castle in the first half of the 17th century, but the owners did not enjoy it for very long. The importance of this defensive building diminished with each subsequent war. At the end of the 18th century, not much remained of the building as it was burnt down by the Russian army in 1664. Only part of the ramparts have survived to this day.

The last owner of the Druja estate was Lithuanian artillery general Prince Franciszek Sapieha (1772-1829), who fictitiously sold Druja around 1824 (most probably he gave it to the administrator of his own estate to a certain Joseph Milosz. The Nobel laureate wrote in Searching for the Homeland:

'Józef Miłosz, born somewhere before 1790, went to the East and entered the service of the Sapieha princes. Here, he served in the administration and, through his diligence, reliability and other additional qualities, he earned the Sapieha prince's trust and boundless faith.[...] According to a tsarist decree, Sapieha, unwilling to accept Russian serfdom, was forced to sell off his estates. Miłosz, whom Sapieha had no reason to distrust, offered to sell off his estates according to a fixed key, for which he received a donation made up by a deed of purchase [...].

After him, the estate was taken over by his son, a doctor, Eugeniusz (d. 1885). The Miłosz family lived in the former Sapieha castle, which Eugeniusz rebuilt after a fire in 1812. In fact, a new mansion was erected, possibly using parts of the Sapiehs' castle. The new building, because of its location and respect for tradition, was then called the Castle. The next owner was the son of Eugeniusz and Emilia Miłosz of the same name as his father (d. 1908), and his brother Józef (d. 1914) married to Maria Kowalewska. His daughter Emilia, in unclear circumstances before 1914, sold off most of the land in Drua including the castle. In the inter-war period, the truncated estate was in the hands of Józef's heirs, probably Maria of Kowalewska Miłoszowa, and the former Miłosz house was used as a barracks for the Border Protection Corps and the residence was rebuilt, adapting it to its new function.

After World War II, the building housed the Druja tuberculosis sanatorium, then a hospital and a health centre. Parts of the courtyard park and one outbuilding have been preserved.

The Druja Miłosz family were relatives of the poet, who wrote about them in his book "Looking for the homeland".

The former residence of the Milosz family stands on the bank of the Dvina River, in a former suburb called Svoboda Sapiezhynska. The building was erected on a rectangular plan, on high basements. The front and garden elevations are decorated with three risalits. The central one, at the front, features a wide staircase. The central one in the garden elevation is closed with a pentagon. The manor house has been rebuilt many times and has lost its historic value.