The National League side confirmed on Thursday that Defoe and the club had "mutually agreed to part ways with immediate effect", two weeks before the start of the new season against Sutton United.
The former England striker, appointed in late March as successor to Neal Ardley, guided Woking to a 10th-placed finish in the fifth tier, winning two of his six league matches in what was his first permanent managerial role.
He also took charge of pre-season friendlies against Sholing and former club Portsmouth before his final match, a 1-0 defeat to Braintree Town.
In a statement posted on Instagram, Defoe said: "Unfortunately, circumstances have ultimately made it impossible for me to continue in the position. My time at Woking FC has been an eye-opening managerial experience and one that will undoubtedly make me stronger for the next challenge in my career."
Notably, his statement thanked club owner Todd Johnson but made no mention of Brown, fuelling speculation that the split stemmed from issues away from the pitch rather than results.
That speculation intensified on Thursday evening when a supporter, responding to a post asking whether Defoe would give an exit interview, claimed Brown "thinks he runs the club, calls all the shots, chose all the triallists, none of which JD wanted" and was "very tight with the owner". Brown responded directly, denying any rift over recruitment.
"Of the vast number of triallists, I did not choose one. Not one," Brown wrote. "Jermain led all recruitment and all triallists. All Jermain's contacts, all linked to him. Every single one. Empty promises? We gave everything that was asked. Everything. I'm sure it'll all come out in due course."
Defoe, who scored 305 goals in 763 senior appearances for clubs including West Ham, Tottenham, Portsmouth and Sunderland before retiring in 2022, previously worked in Tottenham's academy and as part of Rangers' interim coaching staff. Woking must now find a replacement at short notice as they prepare for the new National League campaign.