Hapoel had little to offer as Dinamo raced to a 2-0 lead with goals by Izet Hajrovic and Mislav Orsic in the half hour.
Arijan Ardemi added a second-half brace and a late strike by Armin Hodzic effectively sealed the tie.
It was a dreadful debut for Hapoel's Greek goalkeeper Giannis Anestis, who was at fault for all five goals but Israeli manager Barak Bakhar refused to blame his new signing.
"This was a weak team performance and we showed no character. It creates a small crisis but this team has overcome bigger ones."
With no chance of overturning the deficit in next week's second leg in Israel, Hapoel will play Apoel of Cyprus or Flora Tallinn in the Europa League third qualifying round.
Beitar Jerusalem have already been eliminated from the Europa League after losing 1-2 to Georgia's Chikhura Sachkhere.