This week the JC has located one of them, Fritz Lustig, now 93, who for two years spied on the PoWs from 1943 to the end of the war.
In 1940, like many other German Jewish refugees, he had found himself interned as an enemy alien by the British on the Isle of Man. But three years later, after a spell in the Pioneer Corps, he was recruited to eavesdrop on the PoWs.
Microphones hidden in the lamp fittings of cell-blocks relayed the prisoners' chatter to listening stations where Mr Lustig and his colleagues were secreted.
"We had old-fashioned telephone exchanges and headphones, and several plugs which we plugged into the cells we covered," he recalled. "Each of us listened to two or three cells. We had a turntable, and if we heard anything significant, we cut a record and recorded what was said.
"There were a few security-conscious prisoners who suspected they were being listened to, but most talked quite freely."
From the bugged conversations of German pilots, sailors and soldiers, the British gained insight into enemy military technology or the state of morale in Germany.
"The only significant thing I remember was when the Battleship Scharnhorst was sunk before D-Day," Mr Lustig recalled. "What the survivors said about the sinking was important for the Admiralty."
The eavesdropping on generals at one camp, Trent Park, revealed the extent of knowledge about the Holocaust.
Mr Lustig, who served at two camps, Latimer House in Chesham and Wilton Park in Beaconsfield, said that his team also occasionally recorded details of atrocities – although the recordings could not be used directly in war trials because the bugging operation was "not in accordance with the Geneva Convention". He even met his wife at Wilton Park fellow refugee, Susan Cohn, now 91. "She did clerical work, dealing with documents. But women did not listen in – only men did." They are the parents of BBC journalist Robin Lustig.
At Latimer, the listeners worked in teams of six to ten, in a secret room in an administrative block known as the M-room. "It was so secret that even my wife didn't know what I was doing," Mr Lustig said.
Another listener was Arnost Lederer, the late grandfather of comedienne Helen Lederer, who talks about his covert work at Trent Park in another documentary War Hero in My Family, which is broadcast on Channel Five on Tuesday (May 15).
Big Baba, as she knew him, fled Czechoslovakia for Britain with his wife and two children on the eve of the Holocaust. It was only when confidential files were recently released that she became aware of his intelligence mission. As well as snooping on prisoners' talk, he also went undercover in the camp, trying to prompt them to reveal information.
"If I could see him, I would tell him how happy I was to know he did that work," Ms Lederer said. "To know that he was at least able to be active, to fight back, to do something – that is really important to me."
David Keys, who believes there must have been at least 100 to 200 listeners, hopes to speak to some of those still alive.
"One of the things we are looking at is what intelligence was gained and what happened to it. We can say it certainly it helped the Allies in the air war and the Battle of the Atlantic," he said. "It also provided some very interesting information about the Holocaust. But to what extent did it save lives or shorten the war? We don't yet know.
"I'm also particularly interested to find out whether the listeners thought they were being infiltrated and whether the prisoners twigged they were being spied on."
He also wants to know more about the location of the listening rooms and the equipment used. "If people don't want to be identified, they don't need to appear on film. We are perfectly happy to respect the anonymity of anyone who makes contact with us."
Historian Dr Helen Fry, author of a forthcoming history on the listeners, The M Room, which is due to be published next year, said: "Their role was as important for winning the war as Bletchley. They picked up a lot of information which enabled the Allies to pre-empt German operations."
Mr Lustig's role bugging for Britain did not end with the war: he was transferred to Germany for a year to find out what beans PoWs might spill there.
Contact David Keys either at dmekeys@gmail.com, or phone him or Kate Bullions on 020-7284-6868.
