The German name of the town Wesenberg is derived from the Old High German name for the turtledoves that lived in the area in the Middle Ages. Before the stone castle was built on the natural hill of Vallimägi, there had previously been a wood and earth castle of the pagan Estes called Tarvanpea. Its name meant an aurochs' head in the local language. After it was conquered by Danish crusaders in the mid-13th century, the first stone fortifications and a parish church were erected. In the winter of 1268, a bloody battle took place near Rakvere between the Rus of Pskov and Novgorod and the Livonian Crusaders supported by Danish troops. After the crusaders lost the battle, the castle became a refuge for the survivors. In 1302, the settlement was granted town rights, and from the mid-14th century, after expansion, the fortification became a monastic castle that was the seat of the alderman. In the following years, the fortress and the town were subject to rivalry between Moscow and Sweden, changing hands. Between 1602 and 1605, during the Polish-Swedish War, Polish troops were stationed at the castle for three years. Taking this fact into account, the Wesenberg stronghold can be considered the northernmost outpost of the Polish-Lithuanian Commonwealth throughout history. During the Swedish counter-offensive in 1605, the castle walls were severely damaged. After the Livonian wars, the ruined fortress lost its military significance, was soon abandoned and no decision was later made to rebuild it.

The preserved remains of the medieval fortress testify to the architectural grandeur of the building and constitute one of the most picturesque defensive fortifications of North Inflants. The fortification consisted of a trapezoidal high castle, a large rectangular pre-castle adjoining to the south, and a line of modern fortifications (from the end of the 16th century) encircling the entire castle hill. The high castle represents the model appearance of a conventual monastic building with a two-storey building of four wings around an open courtyard. Presumably the most important rooms were located on the first floor, with communication between them provided by the cloisters. The existing layout of the rooms on the first floor, which was prepared for museum and exhibition purposes (including a chapel, chapter house, etc.), should be regarded as a rather free interpretation. There were three gates leading to the high castle - two external ones, the main one (from the east) and the northern one, and an internal one (from the south), which connected it with the bailey. The defence system of the high castle was strengthened by two corner towers (relatively well preserved) in the south wing and the main tower, the so-called final defence (from the west), which remained as a relic. A historical thematic exhibition was arranged in the high chambers of the castle - including replicas of medieval armour, while in the cellars there is the so-called "torture chamber". The bailey has been arranged as an open-air museum, including a shooting range, smithy and tavern. On the opposite (northern) side of the castle hill, an expressive and monumental sculpture "Tarvas" ("Tur") by the well-known contemporary Estonian sculptor Prof. Tauno Kangro was erected in 2002 to commemorate the 700th anniversary of Rakvere's town charter. The pedestal describing the history of the castle also bears an inscription in Polish: "In the years 1602-1605, Gród belonged to the Polish-Lithuanian Commonwealth." To the east of the castle hill the former town buildings extend. The town was never surrounded by defensive walls and probably did not develop a typical central market square, gradually expanding along the wide main street. Presumably, this axis could have been the contemporary Pikk Street with its carefully restored house facades. The oldest of the town's buildings is the parish church of the Holy Trinity (19 Pikk. St.) dating back to 1254. The preserved Gothic building was built in the 15th century. The church, which is now used as an Evangelical church, contains valuable interior decoration, including a carved Baroque pulpit made by the master Christian Ackermann. On the eastern side, the Kirikupark is adjacent to the church grounds. Behind the bridge over the pond, in one of the park alleys on the right, the person of Prof. Stanisław Herbst (1907 - 1973) is commemorated. The future prominent Polish historian was born into a Polish landowning family in Rakvere. He was a professor at the University of Warsaw and the Military Academy and president of the Polish Historical Society. He specialised in the history of Poland from the 15th to the 18th century and military history, and was also involved in historical geography and art history. On the commemorative stone unveiled in 2008, two granite plaques were erected: one with an image of the professor and the other with an engraved inscription in Estonian and Polish): "PROF. STANISŁAW HERBST BORN IN RAKVERE EMINENT POLISH HISTORIAN - SCIENTIST AND LECTURER AUTHOR OF MORE THAN 600 WORKS, MAINLY ON MILITARY HISTORY, HISTORY OF CITIES AND MODERN CULTURE, E.G.: "INFLACK WAR 1600 - 1602. DURING THE SECOND WORLD WAR HE TOOK PART IN SAVING CULTURAL PROPERTY, SOLDIER OF THE NATIONAL ARMY, ON THE ORDERS OF THE AUTHORITIES OF THE POLISH UNDERGROUND STATE HE DOCUMENTED GERMAN CRIMES AND TOOK PART IN HIDING JEWS."