The mountains are iced with white snow and there is soft blossom on the nut trees. Broad, grey and bouldered, the Gudialcay river rushes east towards the Caspian Sea; a few miles north is the border with Dagestan. This is Azerbaijan, where a hot morning sun glares off the riverside mosque's tin roof. The ancient town of Guba is known for its carpets and its walnut halva. Less known is the Jewish town on the Gudialcay's left bank.

Gyrmyzy Gasaba (in Russian, Krasnaya Sloboda or "Red Roofs") is perhaps the world's only all-Jewish town outside Israel. Where Guba seems poor and perhaps a little care-worn, Gyrmyzy Gasaba appears to be prosperous and thriving - 3,600 mountain Jews live here, wearing their Judiasm with pride, and without fear. "Shalom!" I'm greeted, time and again: "Shalom Aleichem!" Boys in kipot bustle about, and each house displays at least one large Magen David.

Azerbaijan is proud of its tolerance towards minorities. The national religion is Shia Islam, but Azerbaijanis are scorned by their Iranian neighbours to the south as bad, lax Muslims. That Azerbaijan enjoys friendly relations with Israel seems only to prove the Iranian point.

Gyrmyzy Gasaba's people are friendly, too, albeit careful. "Are you here to portray us as anthropological oddities?" asks Rav Adam. He is standing on the yeshivah steps, a slightly stooped, bearded figure, grey-shirted and impressively pullovered despite the pounding heat. I assure him that, as the child of a Sephardi-Ashenazi mixed marriage, I am not.

"We too are Sephardim," he gestures, his movements taking in the grand mansions along Isaac Khanukov street, adding: "But we get on very well with Ashkenazim too, especially those from Lithuania."

Over glasses of black tea in the yehivah's airy vestibule, Rav Adam talks of the mountain Jews' journey to Guba. We sit surrounded by all the signs of the devout life, accompanied by the smells emanating from the fountain to our left where the shammas is scaling and filleting a bucketful of fish. Rav Adam exchanges a few words to him - they speak Juwuro-Tat, a language based on ancient Persian, then seasoned with Aramaic, Arabic and Hebrew.

There are many theories as to who the mountain Jews are, and how they came to be here. One suggests that they are descended from the Khazars (unlikely - the Khazars came later), another that they are ethnic Persian Tats converted to Judiasm; some ethnologists regard Tats as mountain Jews converted to Islam.

Outside the yeshivah, Rav Adam subscribes to none of these schools. "Our people came here from southern Persia around 720 BCE," he says, slowly. "It seems there were some upheavals in what is now southern Iran and Iraq. We were Jewish military colonists loyal to Parthian and Sassanid rulers, sent here to the Caucasus to guard against Mongol invasions from the Pontic steppe."

Mountain Jews settled the eastern Caucasus in towns all over Dagestan, Chechniya and Azerbaijan. "There were times we were persecuted, and times of peace," says Rav Adam. "Arabs came in the 8th century, and there were forced conversions…" The Safavid ruler Nadir Shah was especially cruel, but following medical intervention by a Jewish doctor who saved his son's life, Fatali, the Khan of Guba, granted the mountain Jews sanctuary in his lands. Thus, in 1742 ,was Gyrmyzy Gasaba formally established. "It was known then - and now - as 'Little Jerusalem' and as a centre for Torah learning," explains Rav Adam.

A tumble of teenage boys in knitted kipot arrive. As they remove their shoes to enter the yeshivah, Rav Adam leaves me with young Rabbi Elezar to drive around town. The rabbi (who is also the community shochet) and I pile into his Mercedes and take off at a stately 15 mph. "There were once 11 shuls on this street alone," he says, pausing to return the waves of passing congregants.

All that changed with the 1917 Russian Revolution, when Gyrmyzy Gasaba's Jews were banned from speaking Hebrew or openly practising their religion. "We kept the fires of Judaism burning, but secretly," says Rabbi Elezar. "All the shuls were closed, and this one…" we've arrived at the High Holydays Synagogue: "The Soviets turned it into a footwear and hosiery factory." It was returned to the community in 1991, when Azerbaijan gained independence.

The shul is beautiful, and vastly ornamented with Magen Davids. Inside is a 200-year-old Torah encased in solid silver; the floor is strewn with fine Guba carpets; there are mahogany cubbyholes for congregants' shoes and socks in the chandeliered lobby.

In 1917 there were 18,000 Jews here, but Soviet persecution and attendant famines caused many thousands to flee to Baku, Azerbaijan's capital, and beyond. Between 1979 and 1990 many of these moved to Israel, and some to Russia and the United States. Prosperity followed, and in recent times mountain Jew oligarchs Telman Ismailov and Irmik Abayev have been energetically supporting the rebirth and growth of Gyrmyzy Gasaba. As well as owning local property, retail businesses and agricultural land, mountain Jews are also instrumental in the financial and oil services industries in Baku. Rabbi Elezar is proud to relate that his community now has two shuls, 30 boys and 20 girls studying at the yeshivah, and a mikvah.

Oligarchical and community support is especially evident a five-minute drive away, where finishing touches are being put to the Bet Knesset synagogue, which has air conditioning, and a lighthouse-like roof light from which a huge menorah beams out in all directions. The Azerbaijani taste is for ornamented zinc roofs, and in Gyrmyzy Gasaba this takes the form of metal Magen Davids that sprout everywhere above downpipes, on ceiling overhangs, and in the eaves of the well-kept houses, shops and banqueting suites.

Guba's mosque sits on a high bank directly opposite the Bet Knesset shul, 500 metres across the river. In summer, maariv coincides with the evening call to prayer. How are relations between the communities? Rabbi Elezar beams: "Excellent! Friendly! Respectful!"

I wander past the Jewish pharmacy with the Jewish cabbie sat outside it, past the Jewish photo store and the teahouse where Jewish men are sucking lemon tea through sugar lumps. Rav Adam welcomes me again at the yeshivah, and shows me the 18th-century chumash still in daily use. He beams a smile as broad as the Gudialcay: "This is Jewish life, and we're living it well."