Bridgetown, Barbados. I’m sitting in a cafe in the old town overlooking the historic Garrison and the sweeping Savannah race course, waiting for historian and retired diplomat, Dr Karl Watson. He’s running late, but I’m desperate to meet him, so I’ll wait as long as it takes.

We’ve emailed, before I came to the island, and I believe he can shed important light on the Jewish contribution to West Indies history — and, more importantly to me, the history of my own family on Barbados.

Nearly three decades ago, when I last visited, I was crazy for the beaches, and rum punch at sundown. I didn’t give a hoot about why my ancestors came to the island. That’s changed. This time, I’m determined to uncover a mystery.

You see, most of my family is Jewish. But my maternal grandmother was not. At least, that’s what I thought... Born Nancy Bowring, she grew up in a family of great privilege, at the centre of Barbados white society. As a child in London, I listened rapt to tales of their grand colonial life on the island. A world that has disappeared — of tennis parties, servants and enormous houses.

They played polo and raced horses on the Savannah. My great-grandfather was the first commodore of the Yacht Club and briefly captain of the West Indies Cricket Team. In 1920 they entertained the Prince of Wales during his diplomatic tour of the Caribbean. I never thought they had Jewish roots.

Over the past few years, my mother has become increasingly curious about Nancy’s mother, my great-grandmother, Violet DaCosta. A few years ago, she found graves bearing the name DaCosta in the Barbados synagogue. We’ve done lots of research since then — spent many hours in the British Library — but stalled at the end of the 18th century. Then an email from Dr Karl Watson arrived in my inbox in early July, and the pieces started to fall into place. I knew I had to meet him face-to-face.

Waiting to meet him with me is Miguel Pena, head of the National Trust of Barbados. Miguel has been showing me round the island all morning, on a bespoke heritage tour, learning about how this island, 100 miles east of the other West Indian islands, came to be so important to the British Empire, and to the sugar trade.

In the centre of old Bridgetown, we stopped to look at a very old department store, DaCosta & Co. I’d heard about it many times before, but posing for a photo with my teenage daughter felt very special. The store was founded by my great-great-great-grandfather, David DaCosta, a hugely successful businessman known as the “Napoleon of Commerce”. Records show that he was christened aged five, in 1824, but came from a Sephardic Jewish family.

His own father, Isaac Hisquia DaCosta (1790-1852) was brought up Jewish but left the synagogue after marrying on the neighbouring island of St Vincent.

Why? Well it turns out that Isaac’s wife was a “free coloured woman”, Rebecca — most likely my great-great-great-great grandmother.

I’d discovered this on a genealogical website just before my visit, and wondered how common it was for a Jewish man to marry a black woman in the early 1800s. Was this why Isaac left the synagogue, and had his son christened?

The synagogue is the oldest in the so-called New World, but fell into disuse and had to be saved from demolition by Sir Paul Altman, whose family came to Barbados in 1931. Now painstakingly restored, the synagogue site (including a cemetery and mikvah) is a prized Barbados National Trust property protected by Unesco within the World Heritage Site of Historic Bridgetown.

Beside it, there’s a wonderful museum of Jewish history. Here, guided by the architect Geoff Ramsey, I find DaCosta among other Sephardic names listed abundantly on the restored gravestones. Then I sit and watch a film about how the Jews came to Barbados.

You probably know some of the story already. It starts in Spain, where Jews flourished during the Golden Age of the late middle ages but were forcibly converted after the Moorish rulers were expelled. Many Jews fled to nearby Portugal, only to be followed there by the Inquisition — so they fled again to Amsterdam. And from Amsterdam, many travelled to Recife in Brazil (then owned by the Dutch), looking for new opportunities. When the Inquisition arrived there, the Jews fled again, some finding their way to Barbados.

At my hotel, while others were swimming, I retreated to my room to study my notes. I’d found out a lot in the British Library, in Arnold Wizinitzer’s seminal book Jews in Colonial Brazil. There were references to DaCostas in Recife. Staring over the veranda at the silhouetted heads bobbing in the still waters of the Caribbean, I wondered if those Recife DaCostas might have been my direct ancestors.

According to Wizinitzer, around 150 Jewish families left Brazil in 1654, when the Inquisition arrived there. But some stayed in the New World — founding new Jewish communities in the Caribbean — including Barbados, a British haven recently opened to them by Oliver Cromwell.

It’s intriguing to think I might be descended from these founding refugees, because, despite their small numbers, Jews played a huge role in the island’s early development.

It may not seem so very clever now, but what those Brazilian Jews knew about windmills and how best to extract sugar from cane, revolutionised the sugar trade and significantly boosted production.

There’s a dark side to this. The sugar trade flourished exactly in proportion to the African slave trade, which boomed in the mid-to-late 1600s. From 1627 to 1807, it is estimated 387,000 Africans were shipped to the island against their will. To be perfectly clear: the wealth of the island (and perhaps my forebears) was a direct result of appalling suffering.

By the 19th century, the Jewish community had dramatically declined as the sugar industry faltered, alongside natural disasters and assimilation.

A note in the Jewish museum suggested that many Jewish men married black women, and popular surnames on the island today testify to a Sephardi heritage: Massiah, Aboab, Carvalho, Abrahams, DaCosta, DePeiza, Daneils, Shannon, Pinto and Lindo.

Many (like my ancestor David DaCosta) were assimilated into the white Christian community. It’s hard to establish why, in any particular case, because the records don’t explain. It could have been in preparation for marriage, Miguel suggests, to facilitate overseas travel, or just for social mobility.

Whatever the reason, the once thriving Jewish community on Barbados dwindled to just one observant Jew, who negotiated the deconsecration and sale of the building in 1929. Bevis Marks Synagogue in London acted as trustee taking custody of the Torah, silver breastplate, pointer, cup and candlesticks.

Briefly, the building became headquarters of the Turf Club. My great-grandfather, William — Nancy’s father — was its first treasurer. I doubt he had any idea that his wife Violet’s ancestors had once worshipped there, or that decades later his family would visit its restoration.

Outside the gleaming synagogue there’s a mikvah, excavated in 2008. I’d never been inside a mikvah before, and as I walked down the steps with my mother and daughter something clicked — there are ghosts everywhere. Three generations of DaCosta descendants have returned to a place where our ancestors once bathed more than two centuries before. It’s incredibly moving.

Back at the Garrison, Dr Watson finally arrives, looking dapper in panama hat, white shirt and jeans.

I’ve been hoping for the impossible: that he will tell me everything. Plainly, he can’t. Nobody can. But he sets a large notebook on the table before him and flicks through pages of handwritten pencil notes.

Here, he finds evidence that takes me back one generation before Isaac DaCosta. Isaac’s birth is noted in the synagogue records as 5539 Nisan (25 July 1779), along with his father’s name, Benjamin Henriques DaCosta. Karl also holds copies of various DaCosta wills, the oldest dated 1740.

Wow. I’m so pleased. But there’s still much to do if I’m to trace my family back to Recife. From our email exchanges, I’ve learned that Karl knew my family and my great-grandmother Violet.

His own family have been on the island since the 17th century, brought over as indentured servants. Like others in the poor white community, he grew up in a modest wooden chattel house. His was close to my family’s much grander residences. I ask what he remembers. What was my great-grandmother like?

He looks a little awkward. The former diplomat in him struggles to find something positive.

When he was a boy, he says, eventually, he sold Remembrance Day poppies. At my family’s house, Karl was taken to the servant’s entrance, to be given a coin on a silver plate and a pat on the head from Violet.

He says that visiting their house was like visiting Buckingham Palace — and not, one senses, in a good way.

Well, I never met Violet. But I did know her children, including my wonderful grandmother Nancy, after whom my own daughter is named.

Nancy left Barbados as a young woman and came to London. She met my grandfather, a Jewish doctor, and converted to Judaism at West London Synagogue in 1935.

Did she think of this as some kind of return, to the Jewish faith of her DaCosta forebears? Or was it just chance? Like so much else, I will never know.

Cobbler’s Cove, where Harriet stayed, offers bespoke heritage tours for guests led by Miguel Pena, head of the National Trust of Barbados, and can include the synagogue and excavated mikvah.

www.cobblerscove.com