Instead, it is party supporters such as technology chief Jonathan Kestenbaum, ennobled under his leadership, who have heavily influenced his understanding of the Jewish world.
It was largely the peer's efforts that ensured the Jewish Leadership Council held a private dinner with Mr Miliband in February, allowing him to rub shoulders with the community's biggest machers from the worlds of business, philanthropy and education.
The LFI speech came on the back of his erev-Pesach trip to Israel - one of the first major foreign trips of his leadership. He is a man playing catch-up on all things Jewish, but he is making ground quickly.
"It's almost like he's making up for lost time," a close friend admitted.
His upbringing in north London's Primrose Hill was bereft of Shabbat dinners and he did not celebrate his barmitzvah. That has become a source of great disappointment to him and he has admitted he feels he "missed out" on Jewish youth groups.
The Israel trip had a significant impact. At Yad Vashem, Shoah experts created a special report on the fate of his grandfather. He was deeply moved.
The Holocaust story of Ralph Miliband and Marion Kozak continues to have an effect on him and the community's occasionally aired belief that he falls back on Shoah stories as a type of tokenism is unfair, his friend said.
"He genuinely cares and wants to learn. I don't think his Jewish journey ends when he leaves politics," the source explained.
He leans heavily on Liverpool Wavertree MP Luciana Berger when Jewish issues need a response or statement. And Barrow and Furness MP John Woodcock - not Jewish but a former chair of Labour Friends of Israel - is also said to be another regular source of advice.
Former Labour fundraiser Jon Mendelsohn has also grown closer to the leader since he was ennobled last year. He is regarded as another "go-to" Jewish figure in the party.
Lord Kestenbaum's fellow peer, Maurice Glasman, devised the Blue Labour philosophy aimed at winning back working-class voters. He proposed giving power back to local communities. Originally endorsed by Mr Miliband, it was soon dropped and the relationship between the men suffered.
Earlier this month, Lord Glasman accused Labour of lacking direction. He advised Mr Miliband to read novels rather than think-tank reports during the summer break.
Ironically, it is in his approach to being Jewish that Mr Miliband is most noticeably moving away from looking "weird", the cutting insult aimed at him by critics and shadow cabinet members alike.
Mr Miliband wrote in the New Statesman in 2012: "How can my Jewishness not be part of me? It defines how my family was treated.
"I would not be leader of the Labour Party without the trauma of my family history."
