Your manager should not have singled you and your colleague out. If he sees this as an issue, he should mention it at the start of the next meeting in general terms. He should also be cognizant of his staffs' time, and plan accordingly.