The government has been urged to rethink its plan for a national Holocaust Memorial and Learning Centre in Westminster by a cross-party group of Jewish peers.

In the strongest criticism of the project so far from within the Jewish community, the peers argued in a letter to The Times that the location – next to Parliament - and design “evokes neither the Holocaust nor Jewish history, and the risk is that is purpose will not be obvious to passers-by and it will not be treated with appropriate respect.”

The letter was signed by Lord Sterling, the President of AJEX, Baroness Deech, who has been active in Holocaust restitution, Lord Carlile, the former independent reviewer of terrorism legislation, Conservative peer Lord Grade, Labour peer Lord Haskel, Lord Mitchell, who resigned from the Labour Party over antisemitism, Lib Dem peer Lord Palmer and Lord Turnberg, former president of the Royal College of Physicians.

The £50 million project was initiated by the then Prime Minister David Cameron and the winning design, produced by British architect David Adjaye and Israeli Ron Ara, was announced a year ago by judges who included Chief Rabbi Ephraim Mirvis, Holocaust survivor Sir Ben Helfgott and the Mayor of London Sadiq Khan.

The memorial is planned to be built in Victoria Park, next to the Houses of Parliament, above an underground Holocaust educational centre.

The eight peers said they had all lost close family in the Shoah or been deeply affected by it.

They suggested instead a simpler memorial elsewhere in Westminster, with the rest of the funds used to support existing institutions such as the Holocaust exhibition at the Imperial War Museum.

The project’s intentions would be “undermined it if is erected despite the well-founded protests from neighbours, or if it is seen, quite wrongly, as conveying an impression of national guilt,” they wrote.

In February, another Jewish peer, Lord Wasserman, a close ally of David Cameron, voiced doubts about the project which was approved following a commission led by the former chairman of the Jewish Leadership Council Sir Mick Davis, who is now chief executive of the Conservative Party.

In response to this week’s objections, Sir Eric Pickles and Ed Balls, who co-chair the advisory committee for the memorial, said that although the letter “makes a number of suggestions for alternative sites for the Memorial, these might have been sensibly debated two and a half years ago at the time the announcement to Parliament in January 2016 by the then Prime Minister, selecting Victoria Tower Gardens.”

They added that the site next to parliament “will remind visitors of the importance of democracy in defeating tyranny, and remind legislators that Parliament has the power to protect or oppress its citizens”.

The peers’ letter follows another to The Times last week by a number of organisations and local residents and MP Sir Peter Bottomley, which said the memorial should be in the grounds of the Imperial War Museum in Lambeth because the planned Victoria Gardens site was too small and that the Memorial and Learning Centre, which are anticipated to attract 3,000 visitors a day, “require a bigger, more appropriate site”.

The Imperial War Museum has made clear it would welcome the project in its grounds.

At this week’s Conservative Party Conference, Sir Eric outlined further details of the content of the proposed learning centre, which would cover the Kindertransport of Jewish children from the continent to Britain, the British liberation of Bergen-Belsen concentration camp and  the British internment of German Jewish refugees on the Isle of Man during the war.