Harav Sholom Schwadron, zl, cites a story that occurred concerning the Chafetz Chaim that gives insight to the Patriarch's decision.
The Chafetz Chaim permitted a student who was suspected of transgressing a number of sins to remain in the yeshivah. A short time later, a woman came to complain that one of the yeshivah students who ate his meals at her house was a mechutsaf, insolent. The Chafetz Chaim immediately had the student dismissed from the yeshivah.