Event marshals apparently stepped in to ask the men to leave and police also intervened to shepherd the group away from the crowd.
Police have launched an investigation into the incident after one man had reported being assaulted.
German military clothing from the period is not currently banned - although the North Norfolk Railway, which ran an associated event over the same weekend, banned such garments.
The incident prompted calls to ban Nazi uniforms from future events and the town council has agreed to discuss proposals to make the event Allied-uniform only.
Mike Keller, who lost family members in the Holocaust and witnessed the incident told the Eastern Daily Press: “It was a lovely family atmosphere and very friendly, when suddenly from nowhere there were 10-15 men dressed in authentic SS uniform literally marching in unison.
"It was deeply offensive. These men were not milling about and blending in among people. They were marching and making a demonstration. It was frightening.
"My father was from a Jewish family who lost his parents and brothers and sisters in death camps. He was fortunate to escape with my uncle via Kindertransport, so having to see this with my son was mortally offensive and a disgraceful act."
Karen Pollock, chief executive of the Holocaust Education Trust, told the JC: “It is shocking and offensive to see in 2023 people dressed as Nazis and members of the SS – the very people responsible for the horrific ill-treatment and murder of 6 million Jewish men, women and children during the Holocaust.
“These people should not be glorified – and it sounds like the majority in attendance agree.”
But the men defended wearing the uniforms and said the confrontation was an isolated incident.
A spokesman for the Eastern Front Living History Group said: “We represent the western European nations that fought against Stalin and communism during World War II.
“We were wearing Waffen SS infantry uniforms displaying national shields and insignia of the countries portrayed. Not one member of the group portrayed a German.
“The uniforms were supplied by Germany, as were the weapons, to the foreign volunteers from 1941 to 1945. They’re as close to authentic as you can get.
Sheringham, Norfolk (Photo: Wikimedia Commons)
“As a group we’ve been attending the Sheringham 1940s weekend for four or five years running and never had any problems before.”
They added: “We were walking down Sheringham high street and people were stopping us, shaking our hands and wanting to take photos. It was a brilliant vibe. There was no one upset or offended at all. It was good-natured fun, as it should be.
“As a group we do not tolerate any politics or any form of religious persecution. We simply won’t have it. That behaviour disgusts us and tarnishes what we do. We were not asked to leave. We were leaving anyway to go back to our campsite. We felt it was the right thing to do.”
A statement from the Sheringham Carnival Committee read: “Our volunteer Marshals immediately took action with the help of the Police and our volunteers escorted the group out of the Town.
“We are working collaboratively with Sheringham Town Council, the Police and NNR to reflect on what happened and prevent any future disruption to this event.”
Sheringham Town Council said: "Sheringham Town Council has been made aware that on Saturday there was an incident in the town that was managed by the police.
"Sheringham Town Council will consult with the police, North Norfolk Railway and others to determine what happened and what action may be taken to prevent a recurrence."
A spokesperson for Norfolk Police, said: "A police officer on patrol in Sheringham came across a confrontation in the High Street on Saturday (16 September) at about 5.30pm, involving a man and a group of people who had been attending an event.
"The officer intervened and quickly resolved the incident. One man reported being assaulted and this is being investigated further. No one was injured during the incident."
Reaction from the Sheringham community have been mixed. On social media in a local facebook group, one resident said: "If you dress up as a nazi murderer expect to get called out for such. to be clear, not dresses as a German Soldier but rather wear the uniform of an organisation that killed millions of men, women and children."
Others however defended the men, with one user saying: "Trust the media to blow everything out of proportion. Did they complain about those dressed as the Peaky Blinders? Nasty bunch of vicious gangsters"
