Early clue was South of M20, so certainly not Upchurch!
Allington was a good start, go west (very ish), but not too far.
Breaking news…
Google Maps doesn’t name the castle! And the lane alongside the school no longer reaches our building, that now involves going a road further along and then round the houses, quite literally.
Moral don’t rely on older memories, roads change, the one past the church was a main A road, now it’s 20mph.