Experiential learning using open source is fraught with opportunities for disaster.
Open source is unfamiliar ground for professors. Not the software, which is well-known at this point. But professors aren't accustomed to open development. They treat open source projects like busy work or an easy way to assign homework. Participating in FOSS projects takes work and time, maybe even six months, which is a big commitment for someone who wants to use it in a one-semester course. A student being sent in to a project without involvement from a professor is a recipe for failure.
First, professors need to choose the right projects for their students--not all projects are suited or equipped for an influx of students. And some simply don't want to. Some are not welcoming to newcomers--the kernel is a good example. Projects that have participated in a program for the purpose of welcoming students and newcomers, like Google Summer of Code or OpenHatch, are a good place to start. They're the ones who know how to accept inexperienced contributors.
Ironically, sometimes the problem is that open source communities are too helpful. We understand failure as part of the learning process, and give inexperienced contributors the benefit of the doubt. In a meritocracy, people don't stop you from doing things--they support individuals in their pursuit of a goal, regardless of how wise that end goal might be. If a student says "I want to shoot myself in the foot!" the community is likely to say "Ok! We'll help you aim!" Newcomers have a hard time discerning how much work and skill a project will take, and how big an impact it's likely to have on the users of their code, and helping them constrain themselves to workable problems is particularly important within the tight constraints of a semester schedule.
The second common problem is that professors frequently err by dumping their students into projects as core participants, when by their nature, students should be involved peripherally--which is not the same as invaluable or unimportant ways. Many students don't come with the knowledge of how to make commits to something like the kernel or a major database, nor are they often ready to commit to long-term maintenance. The tasks they work on should be things that are not critical path blockers, but they should improve life for core contributors; ask community mentors what itches they would scratch themselves but are just too busy to do. Then ask if they would teach you how to do that work for them.
Third, students should work on things they already know how to do, to some extent. If students pick tasks that require them to learn a large number of new skills, getting them up to speed on that task may take a long time, which again is a problem if you need to start and stop a project during the course of one semester. Students might spend 12 weeks in a class, 12 hours a week on that class, and half their class time on your project; that's 72 hours total, or less than two weeks of full-time work; think about what an intern or a new employee could do in that timeframe. If you start with something too ambitious, you run into yak shaving. I remember one student group who wanted to fix a project's webpage. Great, we said! But first... they had to learn HTML. No problem. Wait - but first... to check out the webpage's HTML, they needed to learn git. But first... to understand git, they had to learn about version control systems... and so on down a long chain of material seemingly unrelated to their original task. With all that startup cost, students often wonder when they're going to "get around to real work."
Fourth, tell students that the responsibility is on them to understand the level of communication required. Because FOSS projects are widely distributed, they operate only when communication is flowing. Contributors may never meet in person, and if they do, it might only be by chance or once a year at a conference. If communication stops, the project stops. The model students are accustomed to--complete project, submit project, walk away--isn't successful in open source. Look at Mike McGrath's guide on how to become a successful contributor, which focuses almost entirely about communication and culture rather than "technical skill." Learning tools and practices and communicating with a project community is real work - as it is in the "real world." Expect that communication will take 50% or more of your time, and that the actual task you're trying to do will take 33-50%. (Afterwards, Walter Bender said he'd thought these figures were optimistic for seasoned FOSS contributors - after a year or so, you might get somewhere between 30-50%. A newcomer, Bender predicted, would be ambitious to aim for 10%.) So if you're a professor, tell your students that they'll be reading a mailing list every day and keeping an eye on IRC. They should know that merely accomplishing the actual task is a very small part of the overall assignment of interacting with an open source community.
Finally, professors need to get involved in the project they want their students to work on at least six months before a student shows up, and they should be up-front about their plans. That will then give them the opportunity to choose a limited subset of projects that are appropriate to the situation and the class, that have suitable tasks, and that have a community where mentors can be found for the students. Getting involved can be tricky, so if you're a professor new to open source, you might look at Red Hat's POSSE (Professors' Open Source Summer Experience) workshop which introduces faculty to open source communities so they can then introduce their students during the next school year.
There are several additional resources of dedicated people to help you get started: