The downside, as the residents told us, is that if you're missing from a lecture, everyone knows it!
For me, it's an advantage because you get to know everyone really well, and the attendings get to know you. This could be good or bad, depending on what type of person you are and what kind of program you are looking for.