The boxes are piled high in Andrew Dismore's Hendon constituency office. There are unused leaflets, posters and some of the records and documents he has managed to salvage from Westminster. In the middle of the chaos sits a shell-shocked former MP.

He points to the campaign map of the constituency, each district marked with the canvassing returns that he and his small team carried out through the election campaign. The map says he should have won. The map was wrong.

Dismore, Hendon's Labour member for 13 years, lost by the excruciating margin of 106 votes at the General Election to his Conservative opponent Matthew Offord. For the former MP, who describes himself as "a very political animal", the defeat has been a hammer blow. A week after the election, he still seems to be reliving the experience in his head.

"I wake up in the middle of the night with the number 106 going through my head. I think the number will be engraved on my heart until the day I die," he says.

His emotions fluctuate between grief (he describes losing as "like a bereavement"), disappointment and perhaps most of all anger - directed both at his opponent and those who administrated the vote.

I don't want to sound like a bad loser, but basically I was robbed

Such are Dismore's concerns about the way the vote was conducted that he is thinking seriously about a legal challenge.

"I don't want to sound like a bad loser but basically I was robbed. There has been bad administration in every election I have fought in this borough. Until now, it has never made a difference. This time the cock-ups certainly did make a difference.

"There were three recounts. Seventy votes went missing and were never found. There were dozens of people whose postal votes didn't arrive in time. There was a polling station in a Labour district which was misdescribed on the map on the polling card. At one polling station, also in a Labour area, people who were queuing up to vote at 9pm were told not to bother to wait, so they went home."

Of course, the election was always going to be close, but Dismore had been confident of victory right up until five minutes before the declaration, when a pile of late votes altered the balance. "If I had lost by 2,000 it probably wouldn't have hurt this much, but the fact is that I lost by 106 out of a total vote of around 48,000. I have found enough people to think I would have won, had their democratic rights had been respected."

Dismore is also bitter about the campaign fought by Offord. "This is my eighth campaign as a counsellor and an MP and it is the filthiest I've ever been involved in. He was definitely playing the man rather than the ball.

"There were some very personal attacks about allowances. Some of the leaflets were frankly libellous. I've sent them to lawyers. But, actually, the stuff going on below the radar, on the doorstep and on the phones, was even worse."

The personal attacks relate to allegations surrounding "flipping" his two homes and misuse of allowances - charges of which he has been cleared by the Committee on Standards in Public Life.

Dismore is proud of his record as a constituency MP, particularly his work for the Jewish community - detailed extensively in a leaflet which was sent out to Jewish voters. His problems, he feels, were on a national rather than a local level. "People didn't like Gordon Brown. Maybe if we had had Miliband or even Ed Balls, I wouldn't have lost. A lot of people gave that as the reason. The standard response from the Jewish community was: 'We like you but we don't like the Labour government so you've given us a very difficult decision. Why don't you convert to the Tories or go independent?'"

Clearly, some of those Jewish supporters did desert the cause. Dismore also reckons that he lost a crucial number of non-Jewish voters through his support of Israel.

"I spoke out for Israel in the bad times and it cost me Muslim votes. If you are seen as pro-Israel you get a lot of stick. Having the balls to get up in the Commons and defend Israel when the whole of the House is against you is not easy. " Arguably, Dismore's greatest achievement, certainly in Jewish eyes, was his successful campaign to establish Holocaust Memorial Day as an annual event.

"I came from a small town in Yorkshire, but it could have been a small town in Germany. The whole point is to get the wider community aware of the issues. Holocaust Memorial Day also allowed me to establish my credentials. It showed that I was serious about doing something to help, which was good because I think I was viewed with a bit of suspicion at first by the Jewish community."

He campaigned for three years to reform the get law on Jewish divorce, campaigned to defend shechita and secured exemptions for Jewish schools in admissions policy. He is also pleased with the work he did to track Muslim extremists.

He had attempted to expose Muslim clerics, including Bakri Mohammad and Abu Hamza, from 1998. Few if any were interested - until, that is, the 9/11 bombings radically shifted the political landscape. "I had asked 300 parliamentary questions between 1998 and 2001. No one wanted to know. After 9/11 I was the leading expert on extremists in Britain and I was taking calls from all over the world.

"Everyone thought they were just a load of loudmouthed nutters. All my documents have had to be shredded now. When you lose your seat you also lose your data protection exemption. So all the records of my fight against extremism have gone."

If there is one consolation for Dismore it is that the immense amount of work he has had to do in disengaging from Westminster has at least given him a temporary focus. It has been a full-time job just to clear his desk. When the task is finished though, he will have to face life without a seat and without a job - a depressing thought for a self-confessed workaholic.

"I guess the full impact of it all hasn't really hit home yet because I've been so busy clearing out. I have chucked out 80 sacks of confidential material from my Westminster office and I still have 20 or 30 boxes to work through. There are some very nice people there with a very sympathetic bedside manner taking you through the process but the bottom line is they want to get you out so they can get someone else in while the chair is still warm."

Dismore has had no firm thoughts about his next step. Once a high-flying solicitor, he is reluctant to return to the law "because that would feel like going back".

He does not know whether he will fight the seat again but he does intend to continue working for the constituency. Politics is in his blood.

"My grandfather was a local councillor, as was my father. He was killed in an accident on council business when I was 11 and my mother took the seat and kept it until not long before I became an MP. Between us we have 60 years of continuous service. Now it's coming to an end."

Dismore ponders for a moment. "I still feel pretty bruised. And I still feel like I was ripped off."