WeBWorK Workshop[1]

 

 

SIGCSE 2006

 

 

Friday, March 3rd 2006

 

 

 

Olly Gotel*

ogotel@pace.edu

 

Richard Kline*

rkline@pace.edu

 

Christelle Scharff*

cscharff@pace.edu

 

 

Andrew Wildenberg**

awildenberg@cornellcollege.edu

 

 

Jackie Baldwin**

j-baldwin@cornellcollege.edu

 

Eileen Crupi*

eec91084@yahoo.com

 

Tabitha Estrellado*

tabi119@yahoo.com

 

 

 

* School of Computer Science and Information Systems

Pace University
New York, NY 10038

** Department of Computer Science

Cornell College
Mt Vernon, IA 52314

 

 

 

Objectives of the Workshop

 

We are interested in building a community of WeBWorK instructors in the Computer Science field; WeBWorK is currently used to teach mathematics (calculus, pre-calculus, algebra, applied, finite and discrete mathematics) and physics by over fifty institutions. We want to get early feedback from the faculty who will participate in this workshop to inform further development of the project. Participants will also be invited to attend a fundamentally different and more advanced one-day faculty development workshop at Pace University in New York at the end of the project (in May 2007).

 

 

Useful Links

 

WeBWorK Main Web Page: http://webwork.math.rochester.edu

 

Mailing List: webwork-user@gauss.dartmouth.edu

 

Installing WeBWorK2:

http://devel.webwork.rochester.edu/twiki/bin/view/Webwork/WebHome#WeBWorKDocs

 

A Day in the Life of a WeBWorK 2 Professor:

http://webwork.math.rochester.edu/docs/docs/courseadmin/usingwebwork.html

 

Building Problem Sets in WeBWorK 2:

http://www.math.wustl.edu/~blake/wwdocs/admins/building-old.html

 

WeBWorK at Arizona State University:

http://hobbes.la.asu.edu/courses/webwork-help/

 

The PG Language:

http://webhost.math.rochester.edu/webworkdocs/docs/pglanguage/

http://math.webwork.rochester.edu/docs/docs/pglanguage/tutorial/

 

WeBWorK PG Problem Library:

http://webhost.math.rochester.edu/webworkdocs/ww/problemLibrary

 

 

WeBWorK at Pace University, NY

 

http://198.105.44.187/webwork2

 

WeBWorK at Cornell College, IA

 

http://webwork.cornellcollege.edu/webwork/JB04/

<This tutorial is based on WeBWorK 2.2 pre 2 (installed with MySQL 4.0 and CVS 1.12.9)>

 

1.     Introduction: What is WeBWorK?. 3

2.     Extending WeBWorK to Teach Programming Fundamentals. 4

3.     Other Existing Systems. 5

4.     Testimonies. 6

5.     The WeBWorK Student 6

6.     The WeBWorK professor 6

6.1. Course Administrator Perspective. 6

6.2. Managing a Course. 7

6.3. Problem Sets and Problems. 7

6.4. WeBWorK File Organization. 8

6.5. Users in WeBWorK.. 9

6.6. Adding Students and Instructors to a Course. 9

6.7. Creating Problem Sets. 11

6.8. Editing Problem Sets and Problems. 15

6.9. Assigning Students to a Problem Set 18

6.10. Scoring. 19

6.11. Students’ Progress. 19

7.     Future Directions and Feedback from Attendees. 21

8.     References. 22

 

1.   Introduction: What is WeBWorK?

 

WeBWorK (available for full experimentation at http://webwork.math.rochester.edu) is a free open-source web-based assessment system developed at the University of Rochester, under the leadership of Drs. Pizer and Gage to generate, deliver, and (automatically) grade homework problems and distribute their solutions. [11,12,18]. It was developed under NSF grants DUE 9950567 “An Internet-Based System for Mathematics Homework Problems” and DUE 0088212 “An Enhancement of WeBWorK – a Web-based Homework Program”. It is used to teach mathematics (calculus, pre-calculus, algebra, applied, finite and discrete mathematics) and physics by over fifty institutions, including major state universities such as Alabama and Ohio State Universities, liberal arts colleges such as Union and Dartmouth Colleges, and universities in Canada and Asia. WeBWorK’s goals are “to increase the effectiveness of traditional homework as a learning tool by providing students with immediate feedback on the validity of their answers, giving students the opportunity to correct mistakes while they are still thinking about the problem,” and to encourage active learning [19]. There has been promising research on the relationship between online feedback and academic performance [4,5,6,7,22] and the re-testing opportunities to master learning [9], two features offered by WeBWorK. Instructors can select and edit problems from a problem library or write their own customized WeBWorK problem sets (with their solutions, grading schemas and deadlines) in the Problem Generating (PG) macro language that mixes Perl, LaTeX, HTML, and text. PG also permits the drawing of graphs and functions, and it can also be extended to use JavaScript and interface with Java Applets. Problems that can be defined range from true/false and multiple-choice problems to sophisticated matching problems (evaluated via scripts). The power of PG comes from its ability to recognize and substitute formulae (e.g. WeBWorK would equally accept x+1, (x^2-1)/(x-1) or x+sin(x)^2+cos(x)^2 as an answer), and to generate individualized versions of problems using predefined pseudo-random and Perl functions. Instructors have access to statistics that provide “information on the performance of individual students and the course as a whole,” and the system permits them to monitor students’ work [19]. WeBWorK is a successful system liked by students and faculty [19]. A study at Rutgers State University of New Jersey showed a strong correlation between WeBWorK scores and final scores in calculus (where the average rose from C+ to B), especially for first-year students [23]. Similar results were found at Stony Brook University for discrete mathematics [20,21,24]. There are also indications that using WeBWorK for algebra is especially effective with women [13], who represent a minority amongst both mathematics and computer science undergraduates.

 

2.   Extending WeBWorK to Teach Programming Fundamentals

 

Based on the success WeBWorK met in teaching mathematics and physics, we are interested in building a community of WeBWorK instructors in the Computer Science field. We are focusing on developing problem sets for Java, Python and SML, as these languages are used to teach computer science students at Pace University and Cornell College, the two institutions involved in this project.

 

For WeBWorK to truly have a significant impact in those courses spanning the programming fundamentals and more generally in programming, it needs to be able to evaluate programs and program fragments. Our first step is to provide WeBWorK with the ability of evaluate Java program/fragment correctness by interfacing it with JUnit [14], an existing open-source testing framework. JUnit runs various tests to verify that the program/fragment satisfies a stated specification; the number of tests that fail and succeed is provided to the students, and is used to determine the score for the problem. Students are notified of programs that do not compile and terminate (determined by timing the execution of the programs).

 

3.   Other Existing Systems

 

WeBWorK has many advantages with respect to other similar available systems. Popular general courseware systems such as Moodle [16] (free, open-source), BlackBoard [3] (commercial) and WebCT [25] (commercial) are all capable of delivering quizzes. Developed for delivering mathematics homework assignments, WeBWorK is a domain specific environment that allows much more randomization and complexity in the questions and answers. When the answers are (mathematical) formulae, WeBWorK can evaluate if the student’s answer is equivalent to the right answer. CodeLab [8] and OWL [17] are two proprietary web-based programming exercise systems focusing on Java, C++ and C. CodeLab delivers short exercises focusing on a particular programming construct or idea. OWL delivers programming problems where students can enter consequent program fragments that are evaluated automatically. CourseMarker [26] is an interesting proprietary client/server system that grades programming assignments with lexical/typographic analysis and automated testing. It has some extended feedback capabilities including the generation of customized problems for a particular student. However, in these systems, problems may only be selected from fixed (purchased) databases – instructors may not customize problems. These systems have also fixed and per-user costs. Codewitz [15] is a free system focusing on providing animated demos reinforcing core programming concepts, with small quizzes at the end of each. We believe that, with minor modifications, WeBWorK and its (already powerful) problem generating language could address a wider range of problems associated with programming fundamentals than current commercial alternatives at no cost to students and faculty. Our first step was the interfacing of WeBWorK with JUnit.

 

·         CodeLab

http://www.turingscraft.com/

 

·         CourseMarker

http://www.cs.nott.ac.uk/CourseMarker/

 

·         OWL

http://owl.course.com/

http://owl.course.com/owl-c/demo/owldemo.cgi?Mode=2&BookGroupID=Java

 

·         Codewitz

http://www.codewitz.net

 

·         PowerPoint slides comparing the different systems.

 

 

4.   Testimonies

 

 

 

 

5.   The WeBWorK Student

 

Students can do their homework/quiz assignments, download a hard copy of their homework, email their instructors, access their grades or their account, and report bugs.

 

SIGCSE2006 is the name of the course we will consider in this workshop.

Go to: http://198.105.44.187/webwork2/SIGCSE2006/.

Log in (using the username and password provided separately) and go through a sample of Python, Java and SML problems.

You are logged in as a student.

 

 

 

6.   The WeBWorK Professor

 

Go to: http://198.105.44.187/webwork2/SIGCSE2006/.

Log in (using the username and password provided separately).

You are logged in as an instructor.

6.1. Course Administrator Perspective

 

In WeBWorK 2 there is a Course Administrator Perspective, where an admin user can:

 

When adding a course, you can directly add an instructor (recommended). If you do not add an instructor at that time you will have to access your course as an admin. If you have more than one instructor for a course, the first instructor registered will be able to add the other instructors.

 

6.2. Managing a Course

 

WeBWorK 2 has been extended to permit WeBWorK professors to choose problems from libraries. Indeed, lots of professors do not write their own problems but choose them from existing libraries. There exist mathematics and physics libraries for college and high-school education at: http://webhost.math.rochester.edu/webworkdocs/ww/problemLibrary.

 

However, if you want to use WeBWorK to its full capabilities, you will have to write problems from scratch. This requires some understanding of the file organization in WeBWorK and learning the Problem Generating (PG) language. We are developing libraries of problems for Python, Java and SML based on the courses we are teaching at Pace University and Cornell College. We also have permission to develop a libraries of problems based on the book Java Software Solution, Foundations of Program Design by J. Lewis and W. Loftus.

6.3. Problem Sets and Problems

 

WeBWorK professors create problem sets that are composed of problems. Each problem set is described in a set definition file with a .def extension containing an opening date, a due date, a date the answers will be posted, and a list of problems it is composed of. For each problem, the number of points and attempts are specified. The number of attempts permits a way to control how students attempt their assignments. The problems are saved in a directory with the same name as the .def file.

 

Example of a .def file: set0.def

 

openDate = 1/7/00 at 6:00am

dueDate = 1/20/09 at 6:00am

answerDate = 1/21/09 at 6:00am

screenHeaderFile = setHeader.pg # Introduce the pb set

problemList =

set0/prob1.pg, 5, 2 # 5 points, 2 attempts

 

6.4. WeBWorK File Organization

 

When you create a course, a directory with the same name is created. In that directory, let us call it <coursename>, there is a file with a .lst extension (that is a classlist file containing information about the students) and a directory called <templates>. In the <templates> directory, there are .def extension files describing problem sets and directories with the same names that will contain the problems written in PG.

 

You can look at the file organization in Instructor Tools / File Manager. The File Manager offers an extended web interface to manage files to avoid using the Unix command line.

 

6.5. Users in WeBWorK

 

There are five types of user in WeBWorK: admin, instructor, student, global_user and practice user.

 

A unique global_user (with global_user as login and no pre-assigned password) is associated with each course. This represents a guest user who has access to the system to try out WeBWorK problem sets and whose answers will not be saved. (This user still exists for historical and technical compatibility reasons in WeBWorK2.) The notion of guest users has been extended to the existence of so-called ‘practice users’. Practice users are created with bogus social security numbers (e.g. 000-00-000a). A course can have several practice users (with practice1, practice2, etc. as logins and no pre-assigned passwords).

6.6. Adding Students and Instructors to a Course

 

There are two ways to add students to a course depending on the number of students to be added. To add instructors to a course, you need first to add them as students and then change their permission to allow them to be instructors.

 

6.6.1. Adding a Few Students

 

Go to: Instructor Tools / Classlist Editor. The Classlist Editor permits WeBWorK instructors to see the list of users of a particular course, edit and change user information, give new passwords, add, import, export and delete users.

 

 

Check: Add <n> student(s) where <n> is the number of students you want to add.

Click on: Take Action!

You will then enter the information of the students (login, password, id…).

 

Note that initially, the password of a student is StudentID and WeBWorK 2 does not require StudentIDs to be unique.

 

6.6.2. Adding Many Students at Once

 

Go to: Instructor Tools / File Manager.

 

You can create a new classlist file or import, edit and complete an existing file of class information. A classlist file ends with the .lst extension and is saved in the <templates> directory of your course.

 

Format of a Classlist File: classlist1.lst

 

000-00-000a ,PRACTICE1 ,JANE   ,C , , , ,                 ,practice1

111-11-1111 ,Smith     ,Donald ,C , , , ,DASmith@uwyo.edu ,dasmith

111-11-1111 ,Smith     ,Donald ,C , , , ,DASmith@uwyo.edu ,dasmiths

 

The fields are:

 

 

Create a New Classlist

 

Click on: New File and follow the instructions. The name of the file must end with .lst.

 

Importing Students into a Course

 

Go to: Instructor Tools / Classlist Editor.

Click on: Import users from file and select the required classlist file.

6.6.3. Adding Instructors to a Course

 

Go to: Instructor Tools / Classlist Editor.

Select the users you want to become instructors.

Select: Edit Selected Users.

Click on: Take Action!

Change the permission level of the users you want to be instructors to 10.

6.7. Creating Problem Sets

 

In WeBWorK 2 it is possible to import problems from libraries or write problems from scratch.

6.7.1. Creating Problem Sets by Importing Problems from Libraries

 

Go to: Instructor Tools / Library Browser.

 

There are different libraries to choose from. The Problem Library contains a collection of external installed libraries (saved and indexed in the MySQL database). The Local Library contains the problems stored on the local WeBWorK server. The Course Library contains the problems of the course. There is a separate process for libraries from Rochester, Union, etc. They can either appear under Local Problems by linking to their directories in the course templates directory, or extra buttons can appear for them by configuring it to do so in the global.conf file.

 

From the Library Browser screen, it is possible to add problems from a library to a problem set, create a new problem set, and view problems from a library.

 

 

Create a Problem Set

 

You can create a problem set by giving a name to this set (without the set prefix) and clicking on Create New Set in This Course.

 

Adding Problems to a Problem Set

 

Adding problems to the target problem set is done by selecting the target set (Add Problems to Target Set), selecting a library and problems in that library. When you click View Problems, you will be able to view the problems of a specified collection; the problems will be visible the same way as they will be when experimenting with WeBWorK. Update Set finalizes the additions of the problems to the problem set. Note that, when adding problems from a library, the problems are not copied.

6.7.2. Creating Problem Sets and Problems from Scratch

 

To create problems from scratch you will use the Instructor Tools / File Manager.

 

Create a Problem Set

 

Click on New File to create a problem set definition file (with .def extension) in the <templates> directory. Let us call this set set0.def. The name of the set has to begin with “set”. When you are creating a problem set from the File Manager you need to use the “set” prefix. Do not use more than 12 characters for the problem set name.

 

set0.def contains:

 

openDate = 1/7/00 at 6:00am

dueDate = 1/20/09 at 6:00am

answerDate = 1/21/09 at 6:00am

screenHeaderFile = setHeader.pg # Introduce the pb set

problemList =

set0/prob1.pg, 5, 2 # 5 points and 2 attempts

 

Click on New Folder to create a folder set0 that will contain the problems.

 

 

Upload your PG problems in set0. Here prob1.pg is uploaded.

 

 

 

 


 

To finalize the process of the creation of set0, set0.def needs to be imported in the Course Library.

Go to: Instructor Tools / Homework Sets Editor.

Check: Import a single set from set0.def.

Click on: Take Actions!

 

Problems are written in the Problem Generating (PG) language and each problem is saved in a file with .pg extension. PG is a macro language that mixes Perl, LaTeX, HTML and text. PG also permits the drawing of graphs and functions, and it can be extended to use JavaScript and interface with Java Applets. Problems that can be defined range from true/false and multiple-choice problems to sophisticated matching problems (evaluated via scripts). The power of PG comes from its ability to recognize and substitute formulae (e.g. WeBWorK would equally accept x+1, (x^2-1)/(x-1) or x+sin(x)^2+cos(x)^2 as an answer), and to generate individualized versions of problems using predefined pseudo-random and Perl functions.

 

 

http://webhost.math.rochester.edu/webworkdocs/docs/pglanguage/

http://math.webwork.rochester.edu/docs/docs/pglanguage/tutorial/

 

We will now let you create your own Java question (true/false, multiple-choice or matching question) and add it to the problem set setTutorial.

 

 

 

6.8. Editing Problem Sets and Problems

 

Go to: Instructor Tools / Homework Sets Editor.

 

Click on the problem set you want to edit (Edit Problems).

 

 

You can then change the open, due and answer availability dates of the problem set, and mark it as visible or not for students. The assignment type can be specified among: homework, gateway/quiz and proctored gateway/quiz.

 

 

The screen below shows the parameters of quiz assignments.

 

 

You can also edit the setHeader.pg file; this file is used to introduce the problem set. This will appear in the right frame (Set Info section) when practicing with the problem set.

 

# screenHeader.pg

DOCUMENT();

loadMacros(

     "PG.pl",

     "PGbasicmacros.pl",

    

);

BEGIN_TEXT

$BBOLD WeBWorK assignment number $setNumber is due : $formattedDueDate. $EBOLD

$PAR

This is the setHeader.pg file.

$PAR

END_TEXT

ENDDOCUMENT(); # This should be the last executable line in the problem.

 

 

 

You can also delete problems from a problem set (by checking Delete it? and Save Changes), edit a problem (by clicking on Edit it) and order problems. The check box Mark Correct? is used to give full credit to all students if a problem was inappropriate for some reason. The weight and number of attempts for the problems can be also attributed. For an unlimited number of attempts, use “unlimited”.

 


6.9. Assigning Students to a Problem Set

 

A student can only access a problem set if this problem set is assigned to him/her first.

Go to: Instructor Tools / Homework Sets Editor.

Click on: Edit Assigned Users for the considered problem set, check the names of the students to be assigned to the problem set, and Save the changes.

 

 

6.10. Scoring

 

Go to: Instructor Tools / Scoring Tools.

You can score a problem set or several problem sets at the same time. The results including the status of the problems, number of attempts and final scores are saved in a .csv file.

6.11. Student Progress

 

It is possible to monitor the progress of students by observing their results for single problem sets or across all sets. The scores and number of attempts for a problem are stored.

 

 

 

7.   Future Directions and Feedback from Attendees

 

We plan to propose several extensions to the feedback capabilities of WeBWorK designed to benefit both students and faculty.

 

7.1 Extending the Feedback Capabilities of WeBWorK for Students

 

WeBWorK is a formative assessment tool [1,2,10]. We will emphasize this engagement in student self-reflection by extending the feedback capabilities of WeBWorK. We plan to add links to tutorials and dictionaries, hints for the problems, including model answers on problems similar to the one being considered and individualistic feedback. One of our proposed approaches will be to guide students toward additional problems (as created dynamically on the fly), to encourage students to practice with prerequisite material. This will be accomplished by constructing a canonical directed acyclic graph (DAG) of topics of programming fundamentals and by the extension of the Problem Generating language (PG) of WeBWorK.

 

7.2 Extending the Feedback Capabilities of WeBWorK for Faculty using Statistics

 

WeBWorK currently provides extensive activity logs, including information about every attempt to answer every question, but only performs rudimentary analysis of these data. We plan to analyze these data using statistics to provide instructors with timely information on student progress to help guide in-class instruction.

 

 

Please send us your comments, suggestions, ideas etc…

 

 

8.   References

 

[1] Black, P. & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education, 7-74.

 

[2] Black, P. & Wiliam, D. (1998). Inside the Black Box. Phi Delta Kappan. 2:139-144, 146-148. Available: http://www.pdkintl.org/kappan/kbla9810.htm.

 

[3] BlackBoard. Available: http://www.blackboard.com.

 

[4] Bower, M. (2002). The Effect of Receiving the Preferred Form of Online Assessment Feedback Upon Middle School Mathematics Students. CATE 2004, August 16-18, 2002, Kauai, Hawaii, USA, 462-467.

 

[5] Cassady, J., Budenz-Anders, J., Pavlechko, G., & Mock, W. (2001). The Effects of Internet-Based Formative and Summative Assessment on Test Anxiety, Perceptions of Threat, and Achievement. Annual Meeting of the American Educational Research Association, Seattle, USA, April 10-14, 2001.

 

[6] Charman, D.J. & Elmes, A. (1998). Computer Based Assessment (Volume 1): A Guide to Good Practice. SEED Publications, Faculty of Science, University of Plymouth.

 

[7] Charman, D.J. & Elmes, A. (1998). Computer Based Assessment (Volume 2): Case Studies in Science and Computing. SEED Publications, Faculty of Science, University of Plymouth.

 

[8] CodeLab: A Powerful Tool for Programming Instruction. Available: http://turingscraft.com.

 

[9] Covington, M. & Omelich, C. (1984) Task-oriented versus Competitive Learning Structures: Motivations and Performance Consequences. Journal of Educational Psychology, 76(6), 1038-50.

 

[10] FTE (1999). Fair Test Examiner (1999). The Value of Formative Assessment. Available: http://www.fairtest.org/examarts/winter99/k-forma3.html.

 

[11] Gage. M & Pizer, A. (1999). WeBWorK -- Math Homework on the Web. Electronic Proceedings of the Annual International Conference on Technology in Collegiate Mathematics. Availalble: http://archives.math.utk.edu/ICTCM/EP-12/P3/html/paper.html.

 

[12] Gage, M., Pizer, A. & Roth, V. (2001). WeBWorK: An internet-based system for generating and delivering homework problems. Joint Meeting of the American Mathematical Society and the Mathematical Association of America (New Orleans, LA, January 2001). Available:

http://webwork.math.rochester.edu/apizer/Minicourse_Talk/contents.html.

 

[13] Hauk, S, Powers, R., Safer, A., & Segalla, A. (2004). Impact of the web-based homework program WeBWorK on student performance in moderate enrollment college algebra courses. Available:

http://hopper.unco.edu/faculty/personal/hauk/segalla/WBWquan.pdf.

 

[14] JUnit: Testing Framework. Available: http://www.junit.org.

 

[15] Kujansuu, E. & Kulmala, M. 2003. Codewitz: producing interactive elearning material for beginners in programming. In Proceedings of the 8th Annual Conference on innovation and Technology in Computer Science Education (Thessaloniki , Greece, June 30 - July 02, 2003). D. Finkel, Ed. ITiCSE '03. ACM Press, New York, NY, 266-266. DOI= http://doi.acm.org/10.1145/961511.961630

 

[16] Moodle: A Free, Open Source Management System for Online Learning. Available:

http://www.moodle.org.

 

[17] OWL: An Online Web-based Learning. Thomson Learning. Available: http://owl.course.com/.

 

[18] Pizer, A., Gage, M. & Roth. V. (1999,2000). The WeBWorK System. NSF 9950567, NSF 0088212. WeBWorK web site: http://webwork.math.rochester.edu.

 

[19] Pizer, A., Gage, M. & Roth, V. (2000). NSF 0088212 DUE CCLI proposal. Obtained from the authors.

 

[20] Scharff, C. & Wildenberg, A. (2002). Teaching Discrete Structures with SML. Proceedings of Functional and Declarative Programming in Education (FDPE), Pittsburgh, 2002.

 

[21] Scharff, C. & Wildenberg, A. (2003). Supporting Discrete Structures Courses with a web-based tool. SIGCSE 2003 (poster session), February 2003, Reno, Nevada.

 

[22] Sonak, B., Suen, H., Zappe, S. & Hunter, M. (2002). The Efforts of a Web-based Academic record and Feedback System on Student Achievement at the Junior High School Level. Paper presented at the Annual Meeting of the American Educational Research Association (New Orleans, LA, April 1-5, 2002).

 

[23] Weibel, C. & Hirsch, L. (2002). WeBWorK effectiveness in Rutgers Calculus. July, 2002. Available:

http://webhost.math.rochester.edu/webowrkAssessment/article/rutgers2002

and http://math.rutgers.edu/~weibel/ww.html.

 

[24] Wildenberg, A. & Scharff, C. (2002). Oliver: an OnLine Inference and VERification system”, Frontiers In Education (FIE), 2002.

 

[25] WebCT. Available: http://www.webct.com/.

 

[26] Yong, C. and Higgins, C. 2003. Automatically creating personalised exercises based on student profiles. In Proceedings of the 8th Annual Conference on innovation and Technology in Computer Science Education (Thessaloniki, Greece, June 30 - July 02, 2003). D. Finkel, Ed. ITiCSE '03. ACM Press, New York, NY, 236-236. DOI= http://doi.acm.org/10.1145/961511.961594


[1] This work is supported by an NSF CCLI AI grant #0511385 entitled ‘Collaborative Research: Adapting and Extending WeBWorK for Use in the Computer Science Curriculum’.