He was playing Sunday League football in the North East, alongside Bobby Davison, when a chance encounter with a Hartlepool United director opened the door to a career that would eventually take him into management.
The remarkable part, Brown explained to Football Presse, was that two players from the same amateur team went on to become professionals.
“It was a rarity that you used to get one player going into the professional ranks from Sunday League football. But to get two from the same team was unequalled. Nobody's heard of it before, you know.
“And I don't think the scouting network and system in them days was anywhere near what it is nowadays. You can, whether you like it or not, it doesn't matter, you know. All eyes are on everybody. Every game's watched a million times by whoever, you know.
"All these students coming out of university with degrees, they want to get in the game of football, and they have to sit and watch football in front of a TV screen to try and scout out and try and prove what they're worth.
“But back in the day, it was one man and his dog on the touchline. And he could have been just a local from the pub. It could have been your best mate. But it could have been the director from Hartlepool United. And that's what it was for me, you know.”
Brown had already experienced the disappointment of trying to enter the professional game. His trial at Sunderland did not lead to a contract, but he insists it was not a question of failing to impress the club.
“I'd had trials with Sunderland, but I was too old. I was 18 when I had a trial at Sunderland, and I didn't fail the trial. I was just 18, and apprenticeship started at 16, so you were just too old, unfortunately.”
It was a frustrating obstacle for a young player hoping to make his way into professional football. But Brown was still playing, and the opportunity he needed eventually arrived through an unlikely route.
“A director from Hartlepool United was walking his dog on a Sunday afternoon. And Bobby Davison had already made the jump, you know. He'd already gone, and everybody was envious of that, me being one of them.
“So from an outsider looking in, you would probably look at me and say, 'Yeah, he's okay. Sunday League football. What's he like character-wise? Go on, we'll bring you down and have a look at you in the pro ranks.' And that's exactly what the director did.
“He invited me down for a Thursday evening training session with their reserves, and after the training session, I signed a non-contract form, which back in the day tied you to a football club. And I was then playing in a better league with better players, and it made me better. It was as simple as it gets.”
The contrast between Sunday League football and the professional game was not simply about the quality of the players. The pitches themselves could be a test, particularly when travelling to away grounds where the facilities and playing surfaces were less than ideal.
Brown remembers one particular area in South Shields, known as Bent's Park, where the location between the park and the sea exposed players to the elements.
“The pitches that we played on were Sunday League, no doubt about it. But there was one area in South Shields, it has to be said, it was known as Bent's Park. And Bent's Park was a plot of land between the park itself and the sea. So you used to get the best of all the weather, sometimes the worst of it as well.
“But when you found a sunny day, if you couldn't grow grass on them pitches, you were in trouble. You weren't a groundsman if you couldn't grow grass on them. So there were always a decent standard. But then you had certainly away venues that you went to and you were up against it, you know. You were on a school pitch probably that didn't have a caretaker looking after the ground, and all of them things come into play.
“But I thought it was a test of mentality, it was a test of people's character. If you can get through that and become a professional footballer, I'm not saying you must have something, but you must have something coming from Sunday League and going into professional ranks, which is surrounded by very good players.”
That transition brought Brown into a dressing room with players who had already established themselves at higher levels. Hartlepool provided the chance to learn from experienced professionals, while the club's scouting operation gave players outside the traditional pathways an opportunity to prove themselves.
Among the people Brown remembers particularly fondly was Kenny Sutherland, who was involved in identifying talent and was a regular presence at training.
“I always remember Kenny Sutherland. He was the guy that was sort of running the scouting system, but he was also at every training session. So you trained on Tuesday and Thursday nights until you were pro. And he was at every training session that the reserves were doing, the non-league players were doing.
“But then, when you get into the professional ranks, you see him at every training session of the first team as well. So people like that are priceless.
“The amount of people that were there from the time when I was 19, 20 to 25, the players still know each other that well. And Kenny was there and he gave a speech. So he's obviously an intrinsic part of a network or a setup that is going to go places only because of people like him.”
For Brown, the experience of entering professional football was also a lesson in how much a player could learn from those around him. His time at Hartlepool would eventually include five seasons and around 220 appearances, a run of consistency that demonstrated he had earned his place.
But the original breakthrough remained the defining moment: an opportunity offered by somebody prepared to look beyond the obvious and give a Sunday League footballer a chance.
Brown's story is a reminder that football careers do not always begin with an academy contract or a carefully mapped progression through the youth ranks. Sometimes they begin on an ordinary Sunday afternoon, on a local pitch, when the right person happens to be watching.
For the young Brown, that person was a Hartlepool United director walking his dog. The chance he offered would change everything.
Look out for our 5-part series with Phil Brown this week as we discuss his almost 50 years in the professional game.