Lutonman has it, it is Newington Manor.
Lots of stories about tunnels to the church, but that’s a long way. More likely some priest holes or storage for contraband.
I remember doing a fete in the grounds with Graham and his steam railway, early 1960s, and a school friend lived almost opposite. As cubs and scouts we went past it on our way to Monkey Island, which is not an island, nor has any monkeys! Just woods, that had a trig point on top of a high point.
Over to you Lutonman.