Project 2 (Fall 2018)

For full instructions on how to complete the projects for this class, see the full project description. The process as a whole is the same for each of the three projects: what differs is the type of problems your agent will be solving.

For Project 2, it will be solving Problem Set C, which looks like this:

Your grade will be based on three components: your agent’s performance (30%), your agent’s implementation (20%), and your project reflection (50%).

Performance (30%)

Upon submission, your agent will be tested against Problem Set C’s Basic, Test, Challenge, and Raven’s problems. Only its performance on the Basic and Test problems will impact your grade.

Each set is worth half the performance grade (15% of the project each). For each, you will receive full credit as long as your agent is statistically significantly better than random chance. That means that your agent should answer at least 7 out of 12 Basic problems and 7 out of 12 Test problems correctly to get the full 30%.

Performance data will be pulled from the autograder; you do not need to submit anything to Canvas for this.

Implementation (20%)

Second, we will evaluate your implementation directly. Here, we are most interested in revision. We want to see one of three things:

  • Your agent reach perfect performance.
  • Multiple submissions where your agent generally is getting better and better.
  • Multiple submissions where you are clearly attempting to improve your agent, even if you are not successful.

Note that although those sound somewhat objective, there is some subjectivity here. Submitting your agent five times on the final day with slight tweaks to each submission is clearly not as good as submitting each of five consecutive days making more substantive improvements each time. We will look at the rate of submission, the amount of revision, and the improvement to performance to assign your implementation grade.

Your implementation grade also covers novelty. Novelty is not required, but if you come up with a particularly novel solution, the grader may award you points for implementation beyond the 20% available.

Implementation data will be pulled from the autograder; you do not need to submit anything to Canvas for this.

Reflection (50%)

For the reflection, you will write a personal reflection on your process of constructing the agent. In writing this, you should answer the following questions:

  • Provide a narrative of how you approached the project. How did your approach change over time? What modifications did you make?
  • Specifically for your final agent, how does it work? What is its process of solving the problem? How does it represent information about the problem?
  • Specifically for your final agent, describe its performance. How many problems does it answer correctly? How efficient is it? How general is it? Does its performance on the Basic and Test sets differ significantly, or are they about the same?
  • Specifically for your final agent, what are its limitations? What types of problems does it currently answer incorrectly? If it currently answers all or almost all problems correctly, what kinds of problems would it struggle with?
  • For both your final agent and your entire design process, connect your project to human cognition. Do you feel that the nature of your revisions reflect the way a human learns from experiences as well? Do you feel your final agent solves the problems similar to how a human would do so? Why or why not?

Your reflection may be up to 10 pages (excluding references) in JDF format. Any content beyond 10 pages will not be considered for a grade. 10 pages is a maximum, not a target. This length is intentionally set expecting that your submission may include diagrams, drawings, pictures, etc. These should be incorporated into the body of the paper.

If you would like to include additional information beyond the word limit, you may include it in clearly-marked appendices. These materials will not be used in grading your assignment, but they may help you get better feedback from your classmates and grader.

Submission Instructions

Complete your assignment using JDF, then save your submission as a PDF. Assignments should be submitted to the corresponding assignment submission page in Canvas. You should submit a single PDF for this  assignment. This PDF will be ported over to Peer Feedback for peer review by your classmates. If your assignment involves things (like videos, working prototypes, etc.) that cannot be provided in PDF, you should provide them separately (through OneDrive, Google Drive, Dropbox, etc.) and submit a PDF that links to or otherwise describes how to access that material.

This is an individual assignment. All work you submit should be your own. Make sure to cite any sources you reference, and use quotes and in-line citations to mark any direct quotes.

Late work is not accepted without advanced agreement except in cases of medical or family emergencies. In the case of such an emergency, please contact the Dean of Students.

Grading Information

Your overall project grade will be posted to the gradebook in Canvas, along with a comment indicating the breakdown of scores across these three categories. Students whose agents perform in the top 10 in the class (Basic C + Test C, ties broken by Raven’s and Challenge problems) will receive 5 extra points on their final average. Note that if the author of a top-10 agent already received this bonus for a previous project, it will skip to the next person; each student can only earn this once.

Peer Review

After submission, your assignment will be ported to Peer Feedback for review by your classmates. Grading is not the primary function of this peer review process; the primary function is simply to give you the opportunity to read and comment on your classmates’ ideas, and receive additional feedback on your own. All grades will come from the graders alone.

You will typically be assigned three classmates to review. You receive 1.5 participation points for completing a peer review by the end of the day Thursday; 1.0 for completing a peer review by the end of the day Sunday; and 0.5 for completing it after Sunday but before the end of the semester. For more details, see the participation policy.