Knowing everything about any open source project is impossible. If you're going to deal with a large community, you're not going to know all the details. This is unlike teaching courses where everything is black-and-white, and there are plenty of reference texts. If you're going to teach open source, you're going to have to change the way you teach. Rather than a lecturer, you're a mentor.
In these classes, you help guide the search for answers instead of handing them out. And it's OK to tell students that. Simply say, "This is different from other classes you've taken." It's a different learning style, which means they need expectations set so they aren't frustrated in the end.
Similarly, you may learn as much as your students do in the process. You're a co-learner. And that can be intimidating. It's hard to stand up in front of a class and admit you don't know everything you're about to teach. But remember that you know more than you think you do. You know what a good assignment is and what good results look like. You know when a class is going badly. And you know how to ask questions. You have the skills and tools to do this; this teaching is just a little different from the usual process.
Position yourself as a learner to your students. Explain that there may be areas in which you don't know the answers. But what you do know is where to find the answers and how to keep the learning on path. Explain the benefit of life-long learning. Then put them in the instructor role. Have students learn and then teach.
Be opportunistic in your instruction. Conference or meeting happening nearby or affordably? Declare a FOSS field trip and go! If a conference isn't feasible, visit the local user group meetings. Remember that not every such event will be a win. But if nothing else, you'll learn about a group's culture and something that isn't the right approach for your students. Alternately, you can invite FOSS developers into your classroom or host a hackathon at your school. Bring the FOSS to you.
Of course, all of this leads to a degree of unpredictability in your course. What happens if your contact leaves the project or the project has architecture changes or gets forked? What if you discover the scope was too large before you're finished? In general, the answer is to be flexible and work with the community. Find a new contact. Figure out how to make your deliverables work, or figure out where you can continue to fit in the project.
You can avoid some of the problems of unpredictability by planning for it, as strange as that sounds. For example, don't work on anything in the critical path of the project. That means your students aren't under the pressure of the project in addition to their grades, and if something goes awry, the project isn't affected. (The community is unlikely to support your class' involvement in that way anyway.) Start small. Identify several small contributions rather than one large one. You can even have students estimate the time they'll think it'll take. This will also help you avoid schedule creep problems.
When it comes to evaluations, define the rubric when you define the assignment. Create a grading checklist for yourself based on it. You can, however, also separate the grade from whether or not the task was accomplished. Grade on the quality of presentation. Get feedback from the community to help with your evaluation.
POSSE (Professor's Open Source Summer/Software Experience) is professional development for instructors interested in student participation in free and open source software. This post is based on a presentation at POSSE 2013 by Heidi Ellis.