Herzog will lead Israel in the Euro 2020 qualifiers. After an illustrious career in Germany playing for Bayern Munich and Werder Bremen, Herzog, 49, has served as caretaker-manager and assistant-manager of Austria and between 2011 and 2016 he was assistant manager to Jurgen Klinsmann for the US national team.
Herzog was recommended by the Austrian Willi Ruttensteiner, who was appointed Israel FA's Sports Director in June.
Herzog said: "It hasn't looked good lately for the Israel national team but we want to change that. This is an honour and a great privilege and challenge for me and I want to lead Israel to the Euro 2020 finals."
Herzog replaces Elisha Levy who resigned last year after a disappointing fourth placed finish behind Spain, Italy, and Albania in the World Cup qualifiers.
The Austrian is best remembered in Israel for the injury time equaliser he scored from a free-kick for Austria in 2001 in a 1-1 draw at Ramat Gan in a World Cup qualifier. Israel had been set to reach the playoffs for the 2002 World Cup finals but Herzog's goal sent the Austrians through instead.
Former Israeli star Eli Ohana said that Herzog will now have a chance to make up for what he did to Israel in 2001. But former Premier League star Eyal Berkovic was critical of the appointment. "What desire will Israel's fans have to come and see matches when such a hated man and an enemy of the team is standing on the touchline
"Whoever made this decision needs urgent psychiatric attention, I can't think of any other explanation for the appointment. It's a huge disgrace."