The first stone was laid on 2 January 1666. Work was interrupted frequently because the Dutch East India Company was reluctant to spend money on the project. On 26 April 1679, the five bastions were named after the main titles of William III of Orange-Nassau: Leerdam to the west, with Buuren, Katzenellenbogen, Nassau, and Oranje clockwise from it.

In 1682 the gated entry replaced the old entrance, which had faced the sea. A bell tower, situated over the main entrance, was built in 1684 — the original bell, the oldest in South Africa, was cast in Amsterdam in 1697 by the East-Frisian bellmaker Claude Fremy, and weighs just over 300 kilograms (660 lb). It was used to announce the time, and alert citizens, since it could be heard 10 kilometres away. It was also rung to summon residents and soldiers when important announcements needed to be made.

The fortress housed a church, bakery, various workshops, living quarters, shops, and cells, among other facilities. The yellow paint on the walls was originally chosen because it lessened the effect of heat and the sun. A wall, built to protect citizens in case of an attack, divides the inner courtyard, which also houses the De Kat Balcony, which was designed by Louis Michel Thibault with reliefs and sculptures by Anton Anreith. The original was built in 1695, but rebuilt in its current form between 1786 and 1790. From the balcony, announcements were made to soldiers, slaves and burghers of the Cape. The balcony leads to the William Fehr collection of paintings and antique furniture.

During the Second Boer War (1899–1902), part of the castle was used as a prison, and the former cells remain to this day. Fritz Joubert Duquesne, later known as the man who killed Kitchener and the leader of the Duquesne Spy Ring, was one of its more well-known residents. The walls of the castle were extremely thick, but night after night, Duquesne dug away the cement around the stones with an iron spoon. He nearly escaped one night, but a large stone slipped and pinned him in his tunnel. The next morning, a guard found him unconscious but alive.

In 1936, the Castle was declared an historical monument (from 1969 known as a national monument and since 1 April 2000 a provincial heritage site), the first site in South Africa to be so protected. Extensive restorations were completed during the 1980s making the Castle the best preserved example of a Dutch East India Company fort.

The Castle acted as local headquarters for the South African Army in the Western Cape, and today houses the Castle Military Museum and ceremonial facilities for the traditional Cape Regiments. The Castle is also the home of the Cape Town Highlanders Regiment, a mechanised infantry unit.