I was delighted to discover last week that Ken Livingstone had described me in his memoirs as a "minor intellectual". We have crossed swords several times, particularly over his attitude to Islamic extremism. I made a highly critical documentary for Channel 4's Dispatches in the run-up to his election defeat in 2008.

I'm sure he had intended it as a back-handed compliment, but I was in good company: Nick Cohen, Michael Gove, Christopher Hitchens, Oliver Kamm, Melanie Phillips and John Ware were also named in this category.

Mr Livingstone took issue with our warnings of the dangers of totalitarian Islamism and suggested we had collectively become "obsessed" with Islam itself. I can't speak for my fellow minor intellectuals, but to this latter charge I plead guilty – I became fascinated with the richness and diversity of Islamic culture when I lived in Paris in the early 1990s and spent many hours at the Institut du Monde Arabe. I have been hooked ever since.

The memoirs are called You Can't Say That. But they might as well have been called Why I Was Always Right. Over nearly 700 pages he catalogues his battles against the forces of darkness, as he sees them, with never a hint of recognition that he might ever have been wrong. His constant self-justification is particularly unattractive when it comes to his descriptions of his dealings with the Board of Deputies and the Jewish community. His childish refusal to give ground over saying that Jewish Evening Standard reporter Oliver Finegold was acting like a concentration camp guard is just one instance of the inflexibility of his thinking.

His invitation to Yusuf al-Qaradawi, the Egyptian cleric who justified suicide bombings against civilian targets in Israel, was one of the most controversial of his career. In his ignorance Mr Livingstone describes al-Qaradawi as the "leading theologian" of Sunni Muslims, when he is rather one of the leading theologians of Islamist sectarian organisation the Muslim Brotherhood and its Palestinian terrorist offshoot Hamas.

Ken has always been overly impressed by clever people

Mr Livingstone explains that he rejected the view of journalists such as myself and Nick Cohen that he was "appeasing radical Islam" in hosting the sheikh. Instead he turned to minor intellectual (my label), detective inspector Bob Lambert, head of the Met's Muslim Contact Unit, who later reinvented himself as Dr Robert Lambert, an 'expert' on Islamic extremism.

DI Lambert's views were reflected in a Special Branch report that said: "al-Qaradawi has a positive Muslim community impact in the UK. His support for Palestinian suicide bombers adds credibility to his condemnation for al-Qaeda in those sections of the community most susceptible to the blandishments of al-Qaeda terrorist propaganda."

When Mr Livingstone's senior race adviser Atma Singh raised concerns about the Muslim Contact Unit and the al-Qaradawi visit, he was cut off at the knees despite having worked closely with the mayor for many years. When he blew the whistle, the Livingstone machine disgracefully suggested he was a threat to national security.

Bob Lambert, the Islamist copper, completed his doctorate on his own police work when he left the force but soon established himself as a regular Guardian commentator. He also set up the European Muslim Centre at Exeter University, with money from the Islamist Cordoba Foundation and Islam Expo, although he had to issue an apology for the first piece of work there after complaints from councillors in east London and local MP Jim Fitzpatrick who were wrongly described as Islamphobic.

More recently Dr Lambert was exposed by the Guardian as 'Bob Robinson', an undercover officer who exposed violent extremists within the animal rights and environmental movement in the 1980s. He has since publicly apologised to a woman who thought she had a relationship with him but was in fact being used to maintain his cover. At the same time, Dr Lambert has issued a statement to reassure British Muslims that he has not been playing a similar double game with his Islamist friends.

Ken Livingstone has always been over-impressed by clever people. He is clearly in awe of his former aide John Ross, whom he describes in his memoirs as a "workaholic professional revolutionary" and a "statistician of formidable intelligence". His use of Dr Lambert's muddled thinking as justification for his invitation to a Muslim cleric noted for his anti-Jewish, misogynistic and homophobic outbursts is part of a wider problem for a man who wishes to return as London mayor.

There's an old Jewish joke designed to prick the academic pretensions of people like Dr Lambert. "What is a phudnik? A nudnik with a PhD". Dr Lambert is a classic phudnik.

Frankly, I'd rather be a minor intellectual.