Spurs are winless with just two points and two goals from their opening five games, a stark contrast to the optimism generated by improved form under Roberto De Zerbi at the end of last season and close to £400 million spent in the summer market.
Robertson, one of ten new arrivals with Premier League experience brought in to add resolve to the dressing room, told the Stick to Football podcast on Disney+ that last season's struggles are still weighing on the group.
"They've had a couple of really tough seasons in terms of finishes in the league," the Scot said.
"They've had to go into the market in terms of they've spent a bit of money, but I do believe Tottenham are now starting a journey. I don't know how far that journey can go, nobody does, but the determination in that group is, like, you can see last season hurt them a lot.
"And maybe we're still guilty of carrying that weight of last season on our shoulders."
The Scotland captain, who joined on a free transfer from Liverpool, pointed to the mood inside the stadium after a recent home defeat as evidence of how deep that anxiety runs, and said the fanbase has earned the right to feel it too.
"We need to forget. We need to now try and get Tottenham back to where we believe they belong. And the fans are still carrying it as well, and you don't blame them. They've been through such a tough time. We need the first win, and once we get the first win, I think we'll be really good this season."
Robertson revealed Spurs had tried to sign him as far back as January, before he eventually left Anfield as a free agent in the summer.
"I didn't want to leave but Tottenham came in for me. They were bang in trouble at that point. It was never in my plan to leave in January. I had six months left and I had the World Cup. But in January time, it started becoming real that I was going to leave, and that's when I came to peace with it, really. It was always Tottenham - as long as they stayed up! At that point, I didn't quite want to go to the Championship.
"So that was the thing, I ended up then getting invested in Tottenham, watching them and especially when De Zerbi came in, I was fully confident. They showed their interest first, and they really wanted me, and I was happy with how much they wanted me."