Not many conversations linger in the mind more than two decades later, but one I had long ago with a fellow resident of this slot has never faded from memory. It was with Daniel Finkelstein — not yet a peer of the realm — when I was a newcomer to the Guardian’s opinion pages and he was chief of staff to the then Conservative leader, William Hague.

We talked through the politics of the day, including the impossibility of the task he and his boss faced in opposing the then unassailable Tony Blair, but soon we got onto our own political stories. How on earth, I wondered, was he a Tory? How could a Jewish boy like him — like us — sit with that lot?

My puzzlement was less ideological than cultural. I pictured the Conservative party as a tribe rooted in Eton College, in chilly country houses and the rural shires, in families who had — to adapt Ed Miliband’s memorable phrase, contrasting his own roots with those of David Cameron — sat under the same oak tree for 500 years. Surely that was an uncomfortable, alien world for the Jewish child of immigrants?

I explained that, to my mind, it was the Labour party of my father and grandfather, the party that had vied with the Communists for the support of the Jewish East End, the party of Ian Mikardo and Manny Shinwell, that felt like the natural home for Jews like us.

Danny’s response stayed with me. He’s explained it on these pages and elsewhere better than I can, but it came down to the experience of his parents — both victims of the twin tyrannies of the 20th century, fascism and communism. Their experience had left their son with an unshakeable fear of the grand, revolutionary scheme. Such projects usually ended in conflict, violence and murder — with Jews all too often singled out for early and extreme suffering. Better the quiet stability of the safe, steady, even dull suburbs than fiery dreams of making the world anew.

I’ve thought of that conversation often these last few years, as our own society has succumbed to the lure of the sweeping, revolutionary vision. On one level, my side of that chat has aged very badly. How quaint it now seems that I once thought the obvious political home for the British antisemite was the Tory party, with Labour cast as the natural defender of the Jew. It doesn’t look that way now, does it? In just the last few days, leaked emails have revealed that Jeremy Corbyn’s team blocked the suspension of a party member arrested last week on suspicion of inciting racial hatred against Jews. Most Jews’ response is to give a resigned shrug. After all, who can claim to be surprised by that degree of institutional racism anymore?

But if I was wrong about Labour, surely Daniel was wrong about the innate conservatism of the Conservative party. For it is the Tories who have given us Brexit, a revolutionary scheme that rips up past precedent and custom in pursuit of a dream of a pure, clean-slate future — an enterprise long on theory and dogma, and short on practice and evidence.

Except I can’t be too smug about that. Daniel was always against Brexit, partly for the good, small-c, conservative reasons he has held for decades.

It’s me who took a while to see the wisdom in his wariness of wild-eyed, revolutionary fervour. Not that I was ever an enthusiast for burn-it-all-down politics. It’s just I never saw it as much of a danger, at least not here in these damp, perennially stable islands.

All that has changed now. Brexit itself has always been an exercise in doctrinaire dogmatism, oblivious to arguments of practical common sense. But now it’s been joined by the growing suspicion that Brexit, or “real Brexit” as its advocates define it, will never happen, that it’s about to be stolen from the people.

With that suspicion has come all the fear and loathing that revolutionary movements always drag with them: the accusations of betrayal and treachery, the claim that the saintly people have been deceived by a wicked elite, that they have been stabbed in the back. This language strikes an especially gloomy chord in Jews. Our long history has taught us that when the search is on for traitors, for unpatriotic elites or those who have cheated the nation of its birthright — well, such a hunt never ends well for us. The sight of vehement Brexiters dragging effigies of Theresa May and Sadiq Khan through the streets — their heads in nooses — sends a chill down the spine. We know how such a mood can spiral out of control.

I look at the possible scenarios ahead: Brexit delivered, bringing only differing degrees of economic pain; Brexit not delivered, sparking cries of betrayal; a general election pitting this hopeless government against Jeremy Corbyn. Upheaval, crisis, a febrile mood of accusation and suspicion — all of it unleashed these last few years and none of it good. That was the world Daniel wanted to avoid when we ate lunch together 20-odd years ago. He was right to fear it. And yet what’s odd, looking back, is that I don’t think either of us — him or me — ever thought it could happen here. And yet here we are.

Jonathan Freedland is a Guardian columnist