Abraham Levy has a problem with labels. As the spiritual leader of Britain's Spanish and Portuguese congregation, the Sephardim, he rejects the denominational categories that divide the rest of Anglo-Jewry. No Orthodox, Reform, Liberal or Masorti tags for him and his flock.

"I follow the halachah as Sephardim have kept it for 2,000 years," he says. "We are Jews without ideological adjectives. There are only geographical adjectives. I believe that, if we had all continued in this way, world Jewry would be in a much better situation."

These are momentous times for Rabbi Levy. It is his golden jubilee - 50 years a rabbi and always based in the same synagogue, Lauderdale Road, in west London. It is also the moment he has chosen to announce his retirement. He will step down in July when he reaches the age of 73, two years before the end of his contract.

The decision comes after an uneasy few months within the community, following splits over the succession. A favoured candidate, the London-born Rabbi David Bassous, leader of the New Jersey congregation, withdrew in January after the result of a members' ballot to approve his selection was challenged by a group of congregants. Last month, one communal elder claimed that Levy was being "hounded out of office".

Levy himself is reluctant to comment. Speaking this week, he said he bore "not the slightest resentment" towards anyone, adding only that: "It's been a privilege to work for the Spanish and Portuguese community for 50 years."

He refuses to discuss the matter of Rabbi Bassous although there are certain lines you can read between. As he tells me, sitting in the living room of his Maida Vale house before his retirement announcement, he has only ever spent 20 minutes in Rabbi Bassous's company.

If Levy is disappointed at the dissent, his sadness is connected most to its effect upon his community. He does not hide the fact that things are not altogether happy in Lauderdale Road.

In some ways, this is not an unusual situation for him. When he wanted to establish his Naima Preparatory School in 1983 - the first Sephardi school to be opened in the UK in 100 years - the congregation wanted nothing to do with it, so he did it on his own and it is one of his great successes.

(He points out that, post-retirement, he will "continue to work for the school, and for the training of rabbis through the Montefiore Endowment".)

The founding of Naima was not the only problem during his 32 years in office, although he likes not to dwell on the time that his predecessor, Dr Solomon Gaon, the man who held the traditional post of haham ("the wise one") retired early and he himself was not given the title in his place. The rivalry at the time between him and Dayan Pinchas Toledano was well known. Now he says tactfully: "I think the problem was that the two most senior rabbis in the community had their own field of expertise and it was considered best to share the tasks between us."

Levy's was undoubtedly the more important post - frequently he was described as the Sephardi Chief Rabbi - but it was Toledano who became head of the Beth Din while Levy became spiritual leader, and, he adds, ecclesiastical authority, a role enshrined in numerous Acts of Parliament. "I have exactly the sort of duties as the title of haham would have given me," he says. But the fact remains that he was not given the title and it is still a sore point, although he does his best to hide it.

"Things were a little difficult at first, but not so now," he says. He insists he was not hurt, but sitting in the room with us, his wife Estelle corrects him. "Yes, you were disappointed," she says. He shrugs slightly.

His is a congregation that has markedly changed in the 50 years in which he has been a rabbi. Few of the old families, like the Montefiores or the Moccattas, are still active. Many of the members come from the Middle East, not descendants of the people who came to Britain after Oliver Cromwell allowed Jews back to the country and in 1701 established the community's "cathedral" synagogue, Bevis Marks.

Levy was born in Gibraltar. His late uncle, Sir Joshua Hassan was the country's first First Minister, and his elder brother Solomon - known seemingly to everyone he meets in the street as "Momi"-- was the first mayor. A younger brother, James, heads the biggest legal firm there and Levy insists James could have become chief minister if only he had wanted the job. A sister lives in Gibraltar, another is in America. He himself has been a resident on the Rock for no more than six of his 72 years.

When he was a young child, the Levys - descendants of a whole line of rabbis - were evacuated to Madeira. From there, he and Momi were sent to Carmel College, where both fell under the spell of its charismatic founder-head, Rabbi Kopul Rosen. "He asked all the boys what they wanted to do when they left school. I was the only one who said he wanted to be a rabbi." So the obvious next step was to go to Jews' College, from which he went immediately to Lauderdale Road, complete with his semichah. The Kopul Rosen spell was not the only one that exerted its influence over the young Levy. He became a fervent admirer of Rabbi Louis Jacobs, the man many think was cheated of the chief rabbinate at the time of the infamous "Jacobs Affair" in the early 1960s. (Jacobs had said that he regarded the Torah as being "inspired" by God, but not completely dictated by Him. As a result, he was barred from both the headship of Jews' College and from his pulpit at London's New West End Synagogue.)

Levy says now: "There was a great deal I admired about Louis Jacobs, but I did have issues with much of his theology."

When Jacobs retired from the synagogue that was created for him, the New London, he asked the younger rabbi to take over. "I wanted to do that," Levy says. "There would have been no difficulty about its becoming an Orthodox synagogue, which in so many ways it always was. There was separate for seating for men and women. The services need not have changed and any conversion and similar problems could have been sorted out."

The trouble was Levy wanted to remain spiritual leader of the Spanish and Portuguese organisation and his lay leaders would not accept it.

There are many who think he would have been an ideal New London rabbi - or minister, a title that has largely gone into disuse, along with the canonicals, the rabbinical robes, which even Levy, a stickler for tradition, now only reserves for special occasions. But, sitting in his box at Lauderdale Road, he is never without his top hat. Discarding that - the equivalent of the traditional Ashkenazi biretta - would be considered a step too far.

Somehow, hats always play an important place in discussions on Judaism. Levy loves to talk about the family wedding picture in which he has a notable place. "A sea of black hats with one white one in the middle - my Panama."

The black hats bring him on to another important issue - relations between various aspects of Anglo-Jewry. He says he gets on very well with the Charedim, although they concern him. "I don't like the idea of so much emphasis being put on their clothes. One's religion should not be decided by the material you put on your head." He himself wears a small black kippah. And he adds, revealingly: "I do not approve of people who put ritual ahead of ethics. Judaism is a combination of ritual and ethics. We need both."

Having said that, he and his community have been affected in recent times by what he considers a lack of respect from other religious bodies, which tend to look down on the men from Bevis Marks and Lauderdale Road. For years, it was the Sephardim who were accused of being snobby, considering the more recently arrived Ashkenazim as upstarts. Today, in matters such as kashrut and conversions, the tables have turned as the non-Sephardi Orthodox consider themselves holier than all others.

As Levy said in a foreword for the Jewish Year Book: "I don't see the right level of respect in Anglo Jewry and I don't see it in Israel either… I have been suggesting for a long time that the different kashrut authorities should ideally merge, the same should happen with conversions."

But the real fault is not laid at the men in black hats. He says: "We get on very well with the ultra-Orthodox. This is very much a game of the United Synagogue - to be as influential as possible in Anglo-Jewry. I keep on saying to the Chief Rabbi that they, somehow or other, have to have the upper hand in everything. I keep saying it is God who has to win. Let's work together."