As ever, Wikipedia is your friend:
https://en.wikipedia.org/wiki/Sutton_Valence_Castle The page on Sutton Valence itself is also worth a read.
The castle appears to be late 12C, so a century after the conquest. Building would be under the control of a master mason who at this date was probably Saxon or Anglo-Norman. Journeyman masons would work under his control for the stone preparation and a "grunt" workforce of peasants and animals would provide the labour, lifting and motive power. BTW, slavery
per se has not existed in England since 1066, peasants were tied to the land, not owned by their lords and had access to manorial justice. We might find the conditions little different from slavery, but they were theoretically legal persons and not chattels.