At the end of the Dutch period at the Cape the Castle with its armament and moat still presented such a display of power and solid impregnability that no attempt had ever been made against it.
Today the Castle, separated from the shore by a large expanse of reclaimed land, still serves the country as the headquarters of Western Cape Province Command.
In 1666 the foundation stone of a more permanent building, the Castle was laid and some twelve years later the whole garrison moved to their new quarters.
By the late 1650s the settlement around the Fort De Goede Hoop had expanded, and also included free burgher farms along the Liesbeeck River to the east of the fort and Table Mountain, cutting off the local Khoi from their traditional grazing areas.
During the early 1650s the dominant Goringhaiqua people claimed grazing rights over the whole Cape Peninsula, but shared it with the Gorachoqua and the stockless Goringhaicona (Bottaro, J. It came therefore as a shock when in 1652 the Dutch established a foothold in Table Bay, erected a basic four-square fort with four hornworks, and some years later proceeded to grant freehold farms to freeburghers along the Liesbeeck River valley.
Two days after van Riebeeck arrived in Table Bay to establish a provisioning station for the VOC ships a gathering of the first ' Council of Policy' decided to begin immediately with the erection of a fort, to be known as Fort De Goede Hoop to house the garrison and government offices.
The Castle at it presented itself then was reported as too vulnerable and given a new couvre-face battery, named Imhoff Battery, between it and the receding beach.This battery was destroyed when the railway line was extended into central Cape Town.