Former prime minister Lord Cameron defended plans for a Holocaust memorial centre right next to parliament in Victoria Tower Gardens.

Speaking at the second reading of the Holocaust Memorial Bill in the House of Lords last night, he told parliament: “I announced the Holocaust Commission back in September 2013. It was multi-faith, with teams of experts, and one of the biggest-ever gatherings of Holocaust survivors. It was also, of course, thoroughly cross-party – anything that can bring together Ed Balls and Michael Gove is worthy of note. It was a genuine attempt to look at this, and it was clear."

He went on to say: “It did not say that the existing memorials are sufficient. It did not say that the current state of Holocaust education was good enough. It did not say that we could put this thing somewhere else in London. It said that there is real power in bringing together the monument and the education, and having it at the heart of our democracy. I want unashamedly to put my cards on the table and say that I think this is the right idea, in the right place and at the right time.”

Baroness Deech, a longstanding critic of the plans to house the memorial centre in Westminster, tabled an amendment to the bill, which was defeated, claiming that it “fails to allow for a full appraisal and consultation on any preferred site for a Holocaust Memorial and Learning Centre”.

She told her fellow peers: “The irony of the Westminster location is that this is the very area where hate-filled marches have taken place for weeks, the police being unable or unwilling to stop them; where politicians have been unable to protect Jewish students from abuse and do not shy away from undermining protection of the land where the Holocaust survivors took refuge. Westminster: where misinformation in the media spreads hate uncontrolled. A new learning centre here would be a model of complacency; an excuse for those who call themselves non-racists to pose by it; a defence against excessive anti-Israelism.”

Concerns over a Westminster memorial becoming a target for anti-Israel activity were echoed by Baroness Fox.

“Let us imagine today visitors emerging from the learning centre and looking up from Victoria Tower Gardens to parliament,” she said.

“What would they see, if we were being honest? This week, they would see a betrayal – British politicians attempting to disarm the Jewish nation after it suffered the worst act of antisemitic barbarism since the Holocaust. Turn the gaze the other way: I worry that politicians will look out to Victoria Park Gardens at this new memorial and conclude, complacently, ‘We built that. It proves that we’re fighting antisemitism and, what’s more, we’re now stamping down on far-right bigotry.’

“So dazzled by its own creation, parliament will turn a blind eye to the tens of thousands of progressives carrying placards featuring swastikas defiling symbols of Israel, or turn a deaf ear to the ugly pro-jihadist, antisemitic chants in the Westminster vicinity. There is a lot more to fighting antisemitism than props. Finding a fitting memorial and a proper way of teaching and learning is not contained within this proposal.”

However, the location next to parliament was welcomed by others.

Former pensions minister Baroness Altmann, herself a daughter of Holocaust survivors, said she was “grateful” to the new government for pushing forward the legislation and welcomed the decision to “leave a legacy mark to demonstrate the concerns about any of this happening once more”.

The Holocaust Memorial Bill was introduced in the last parliament by the previous Conservative government.

However, it did not progress and was one of the few items of legislation “carried over” into the new parliament.

Speaking for the government, Lord Khan told the House of Lords: “Memorials alone cannot prevent antisemitism, but this memorial will play a part in reminding everyone where antisemitism can lead. It will be a reminder to us all, in parliament and across the whole nation, of the potential to abuse democratic institutions to murderous consequences, and it will challenge us to stand up and combat racism, hatred and prejudice wherever they are found.”