But he says that in all the time he was patrolling the grounds with his dog Shadow, there was only one minor vandalism incident before the recent attack which saw the desecration of more than 50 graves.
"It was only a matter of time once I had gone that something like this would happen," he said as a mass clean-up operation began.
"It's a dangerous area. I've been threatened with knives and violence. For years I protected that cemetery and I feel upset and disgusted for the poor families who have people buried there.
"I support the Jewish community but it needs to look after its graves and no one should be treated the way I was. It's no good coming to do a cemetery clear up, just throwing the dumped rubbish in a fire."
A friend, who did not want to be named, said: "It's only been a year since he left and £100,000 worth of damage has been caused. That's more than the entire amount Gary was paid during the 14 years he was working."
The incident is doubly frustrating for Mr Colrein who was forced to live in a single room when the caretaker's house became dilapidated.
He said his complaints fell on deaf ears and there was not even enough money to buy a new shovel to dig graves after burial fees stopped including funds for maintenance.
Meanwhile, police blamed their bungle on human error. An officer keyed the wrong address into the crime logging system. As a result, a car was sent to Manchester General Cemetery on Rochdale Road, two miles away.
But this also meant there was no information properly logged against the correct crime scene, such as calls made at the time from witnesses.
The system's inability to cross-refer such information meant that police had to start from scratch in looking for evidence - including random visits to people who had given details at the time.
Police and Crime Commissioner Tony Lloyd said: "The system is outdated and not fit for modern purpose. The whole purpose of logging a crime is to cross reference incidents. We are spending £35 million on a new ICT system. "
On Sunday more than 100 volunteers, including Christians and Muslims, cleaned the cemetery and helped police search for evidence.
A spray can was seized, thought to be connected to swastikas daubed some weeks ago in a separate incident. Antisemitic insults were also scratched on to stones.
The Rev Eddie Roberts brought volunteer congregants from Sunday services at his St Peter's and St Paul's church.
Ray Isaacson, 69, from Bury, described the attack as a "tragedy" as he came to check on his family graves.
Loading a wheelbarrow of junk and foliage on to a pile of old furniture and carpets, he added: "Following the wrong information doesn't show the efficiency of the police force considering the high risk faced by the Jewish community. But we also have to admit we have neglected our cemeteries for years, never paying for upkeep."
Trust treasurer Stephen Niman said of Mr Colren's living conditions: "If Gary would notified us sooner we would have rectified his living conditions. We did replaced the boiler to the cost of £3,000. But it was up to him to notify us of his living conditions.
"Gary was dealt with fairly and appropriately, but we came to the decisions didn't need a sexton on site. We don't believe a caretaker would have prevented the vandalism. One person cannot cover the entirety of a 15 acre site."
Two 13-year-old youths have been arrested on suspicion of racially aggravated criminal damage.
