Langdon College has been downgraded from "good" to "requiring improvement" in its latest Ofsted inspection.
The college, which is based in Manchester, is run by educational charity Langdon as both a residential and day college for teenagers and young adults between the ages of 16 and 25 with mild-to-moderate learning needs.
Inspectors, who visited the college in both January and March this year, judged the college's leadership, quality of teaching, outcome for learners and provision for learners with high needs as "requiring improvement" - an Ofsted rating of three.
"The quality of learning is not consistently good and does not provide sufficient challenge for all learners," the inspection service's report said. "The arrangements for managers and tutors to monitor and review learners' progress from their starting points are not effective in English, mathematics, independent living, community and work skills."
The college was also criticised for its failure to review the impact of its work on pupils who graduated and for not recording their destinations upon leaving.
Nevertheless, students' "personal development, behaviour and welfare" was rated "good".
Learners were said to demonstrate increased confidence and skills in managing their behaviour.
Safeguarding was also judged as effective, with staff judged to carry out "very thorough risk assessments for learners, covering all aspects of their programme, particularly community activities".
In order to improve, inspectors said, the college must "monitor closely learners' progress towards meeting their targets across all aspects of their programme".
Teachers were also advised to improve their planning and assessments, as they did not "take sufficient account of individual needs".
"Of course, this is disappointing," said the college's former chair of governors, Joy Wolfe. "But we are looking on the positive side - that Ofsted has given us the opportunity to make the necessary improvements.
"We have already addressed and continue to address many of the issues that have been raised, and we have an ongoing discussion with inspectors to ensure we are on the same page."
She added: "The main outcome is that our students come out of Langdon as different people to how they came in. It is still the most amazing place with the most amazing outcomes for our students. We stand by and are very proud of the life skills they develop."
The specialist college currently has nine students on its Manchester site. The charity is planning to open a sister college in Edgware, north London, in September.