There are four sets of problems for your agent to answer: B, C, D, and E. Each set contains four types of problems: Basic<\/a>, Test, Challenge<\/a>, and Raven\u2019s. You\u2019ll be able to see the Basic and Challenge problems while designing your agent, and your grade will be based on your agent\u2019s answers to the Basic and Test problems. Each project will add a problem set or two: on Project 1, your agent will answer set B; on Project 2, your agent will answer sets B and C; and on Project 3, your agent will answer sets B, C, D, and E. Thus, Projects 1 and 2 build toward Project 3, the ultimate deliverable. Your grade will be based on three components: how well your agent performs on the problems, how your agent is implemented and revised, and a project reflection you turn in along with your agent.<\/p>\n Different problems will also give your agent different amounts of information. Certain problems in problem sets B and C (specifically, the Basic and Test problems) will provide \u201cverbal\u201d representations. Verbal representations are structured representations that verbally describe what\u2019s inside \u00a0a figure in a problem. For example, in the first problem below, a verbal representation would describe figure A as \u201ca large, unfilled circle inside a very large, unfilled circle\u201d, and figure B as \u201ca small, unfilled square inside a very large, unfilled circle\u201d, using a more structure representation. Your agent would take those descriptions and produce an answer from the eight choices. In all the other problem sets, however, your agent will only<\/i> be given the images themselves — what we call a \u201cvisual\u201d representation — in .png format. It will have to take in the image of the figures themselves and do its reasoning based on those.<\/p>\n Every problem set provides visual representations, so you can try approaching these problems using visual representations (instead of or in addition to using verbal representations) as early as you want. Project 3\u2019s problem sets (D and E) only<\/i> provide visual representations, so you\u2019ll have to try a visual approach eventually. However, verbal approaches tend to be somewhat easier because a human has already interpreted the figure, so you may find it best to rely mostly on the verbal representations for the first two projects. Note that all<\/i> the optional problems (the Challenge and Raven\u2019s problems) only provide visual representations, so if you want to try those problems during Projects 1 and 2, you\u2019ll want to try a visual approach then. Your agent will run against every problem on Project 3, though, so you\u2019ll never miss out on the chance to give those a try.<\/p>\n Don\u2019t worry if the above doesn\u2019t make sense quite yet — the projects are a bit complex when you\u2019re getting started. The goal of this section is just to provide you with a high-level view so that the rest of this document makes a bit more sense.<\/p>\n This section covers the learning goals and background information necessary to understand the projects.<\/p>\n One goal of Knowledge-Based Artificial Intelligence is to create human-like, human-level intelligence. If this is the goal of the field, then what better way to evaluate intelligence of an agent than by having it take the same intelligence tests that humans take?<\/p>\n There are numerous tests of human intelligence, but one of the most reliable and commonly-used is Raven\u2019s Progressive Matrices. Raven\u2019s Progressive Matrices, or RPM, are visual analogy problems where the test-taker is given a matrix of figures and asked to select the figure that completes the matrix. Examples of 2×2 and 3×3 RPM-style problems are shown below.<\/p>\n In these projects, you will design agents that will address RPM-inspired problems such as the ones above. The goal of this project is to authentically experience the overall goals of knowledge-based AI: to design an agent with human-like, human-level intelligence; to test that agent against a set of authentic problems; and to use that agent\u2019s performance to reflect on what we believe about human cognition. As such, you might not use every topic covered in KBAI on the projects; the topics covered give a bottom-up view of the topics and principles KBAI, while the project gives a top-down view of the goals and concepts of KBAI.<\/p>\n The full Raven\u2019s Progressive Matrices test consists of 60 visual analogy problems divided into five sets: A, B, C, D, and E. Set A is comprised of 12 simple pattern-matching problems<\/a> which we won\u2019t cover in these projects. Set B is comprised of 12 2×2 matrix problems, such as the first image shown above. Sets C, D, and E are each comprised of 12 3×3 matrix problems, such as the second image shown above. Problems are named with their set followed by their number, such as problem B-05 or C-11. The sets are of roughly ascending difficulty.<\/p>\n For copyright reasons, we cannot provide the real Raven\u2019s Progressive Matrices test to everyone. Instead, we\u2019ll be giving you sets of problems — which we call \u201cBasic\u201d problems — inspired by the real RPM to use to develop your agent. Your agent will be evaluated based on how well it performs on these \u201cBasic\u201d problems, as well as a parallel set of \u201cTest\u201d problems that you will not see while designing your agent. These Test problems are directly analogous to the Basic problems; running against the two sets provides a check for generality and overfitting. Your agents will also run against the real RPM as well as a set of Challenge problems, but neither of these will be factored into your grade.<\/p>\n Overall, by Project 3, your agent will answer 192 problems. More on the specific problems that your agent will complete are in the sections that follow.<\/p>\n Historically in the community, there have been two broad categories of approaches to RPM: verbal and visual. Verbal approaches attempt to solve RPM based on verbal representations of the problems. In these representations, a human initially describes the contents of the figures of a problem using a formal vocabulary, and an AI agent then reasons over those representations. Visual approaches, on the other hand, attempt to solve RPM based strictly on the images themselves: they take as input the raw image data and perform their analysis from there. Examples of verbal approaches include Carpenter, Just & Shell 1990<\/a> and Lovett, Forbus & Usher 2009<\/a>. Examples of visual approaches include Kunda, McGreggor & Goel 2013<\/a> and McGreggor & Goel 2014<\/a>.<\/p>\n Much research has been done examining the differences between these approaches in humans (e.g. Brouwers, Vijver & Hemert 2009<\/a>). Within artificial intelligence, visual approaches are generally more robust in that an initial phase of human reasoning is not necessary. Thus, the ultimate goal for these projects will be to solve RPM-inspired problems visually. However, visual problem-solving tends to be significantly more difficult. Thus, you will start by having the option to use both verbal and visual approaches (using verbal representations we have produced for you), and by the last project you will use only visual methods.<\/p>\n This section covers the more specific details of the projects: what you will deliver, what problems your agents will solve, and what representations will be given to you.<\/p>\n In this offering of CS7637, you will complete three projects:<\/p>\n Each problem set consists of 48 problems: 12 Basic<\/a>, 12 Test, 12 Raven\u2019s, and 12 Challenge<\/a>. Only Basic and Test problems will be used in determining your grade. The Raven\u2019s problems are run for authenticity and analysis, but are not used in calculating your grade. Note that each project will also be run on the previous projects’ problem sets, but not<\/em> for a grade: this is just for you to see how your agent improves over time. So, Project 3 will run on all 192 problems (48 Basic, 48 Test, 48 Raven’s, and 48 Challenge), but only the 48 problems in sets D and E will be used for your grade.<\/p>\n On each project, you will have access to the Basic and Challenge problems while designing and testing your agent; you will not have access to the Test or Raven\u2019s problems while designing and testing your agent. Challenge and Raven\u2019s problems are not part of your grade, though note that the Challenge problems will often be used to expose your agent to extra properties and shapes seen on the real Raven\u2019s problems that are not covered in the Basic and Test problems.<\/p>\n As mentioned previously, the problems themselves ascend in difficulty from set to set. Additionally, only visual representations will be given for problem sets D and E, so for project 3 you\u2019ll be required to do some visual reasoning.<\/p>\n For each project, your code must be submitted to the autograder by the deadline. However, it is okay if your project is still running after the deadline. You are only permitted 10 submissions per project.<\/p>\n On each project, you will also complete a project reflection describing your process of constructing your agent, your agent’s performance and limitations, and your agent’s connection to human cognition.<\/p>\n Your grade on the project is based on three criteria: your agent\u2019s performance on the Basic and Test problems (30%), your agent’s implementation and revision (20%), and the project reflection you submit along with your agent (50%).<\/p>\n For performance, your agent will be graded on whether it meets minimum performance goals. The minimum performance goal is: 7 or more problems correct on each set of 12 problems. The Basic and Test sets are each worth 15%: meeting the minimum goal on both earns the full 30%, while meeting the goal on only one earns 15%. Meeting the goal on neither earns 0%. (For Project 3, where there are four sets instead of two, each is worth 7.5%.)<\/p>\n Note that we will use your best submissions to calculate these scores. Your best scores on each set do not need to come on the same submission. For example, if you reached 7\/12 on the Basic B set on your third submission and 7\/12 on the Test B set on your fifth submission, but never reached 7\/12 on either set on any other submission, you would still receive full credit.<\/p>\n For implementation and revision, we will look at the implementation of your agent as well as your patterns of revision. We expect you to run your code on the server relatively often<\/strong>: <\/strong>this is how we will judge your revision process. This area is more subjective, but generally, you will receive credit as long as (a) your agent reaches perfect performance on Basic and Test problems, (b) your agent gets better over a series of a few submissions, even if it never reaches perfection, or<\/strong> (c) you keep trying and making non-trivial revisions, even if your agent’s performance is not getting better. In other words, you’ll receive credit for a successful agent or<\/em> continued attempts to build a successful agent.<\/p>\n You should notice that between Performance and Revise & Reflect, there is no heavy incentive to design a near-perfect agent at all costs. This is intentional: we want you to design agents that are somewhat successful, but beyond that, we care more about novel and interesting approaches than about success-at-all-costs approaches.<\/p>\n For the journal, you will write a personal reflection on your process of constructing the agent. It will include an introduction and conclusion, as well as a journal entry for each individual submission to the autograder. In each journal entry, you should include the submission to which the entry corresponds, a description of what has changed, a reflection on the relationship between your latest agent and human cognition, and a reflection on the agent’s performance.<\/p>\n Revise & Reflect and Journal credit are deeply intertwined: you’ll know that you’re fulfilling our goals for implementation if you find you have plenty to write about in your journal.<\/p>\n As noted, project grading specifically encourages either having a great agent or continually trying to improve your agent. Someone whose agent works perfectly on their first submission will receive the same implementation score as someone whose agent slowly gets better over a series of several submissions, even if they never come close to perfect performance.<\/p>\n However, we do want to have some incentive for doing well. Thus, on each of the three projects, the top 10 performers in the class will receive 5 extra points on their final average. This effectively means they can skip a test (or a couple questions on a homework, or some participation), or just let the extra points compensate for lost points elsewhere in the class.<\/p>\n Note that a single student can only receive this once; students who already received this bonus on an earlier project will be skipped in finding the top 10 on a later project. Top 10 will be calculated by total score on the problems graded for that project (e.g. Basic B and Test B for Project 1; Basic C and Test C for Project 2). If necessary, ties will be broken first by performance on the Raven’s problems, then on the Challenge problems.<\/p>\n To make it easier to start the project and focus on the concepts involved (rather than the nuts and bolts of reading in problems and writing out answers), you\u2019ll be working from an agent framework in your choice of Python or Java. \u00a0You can get the framework in one of two ways:<\/p>\n You will place your code into the Solve method of the Agent class supplied. \u00a0You can also create any additional methods, classes, and files needed to organize your code; Solve is simply the entry point into your agent.<\/p>\n As mentioned previously, in project 3, your agent will run against 192 problems: 4 sets of 48 problems, with each set further broken down into 4 categories with 12 problems each. The table below gives a rundown of the 16 smaller sets, what will be provided for each, and when your agent will approach each.<\/p>\n Key:<\/p>\n Thus, for the first two projects, you\u2019ll be addressing the easier two sets of problems using visual and\/or verbal representations. For the final project, you\u2019ll address the final two sets of problems using their visual representations only. It might, therefore, be prudent to get an early start on the visual methods!<\/p>\n Within each set, the Basic, Test, and Raven\u2019s problems are constructed to be roughly analogous to one another. The Basic problem is constructed to mimic the relationships and transformations in the corresponding Raven\u2019s problem, and the Test problem is constructed to mimic the Basic problem very, very closely. So, if you see that your agent gets Basic problem B-05 correct but Test and Raven\u2019s problems B-05 wrong, you know that might be a place where your agent is either overfitting or getting lucky. This also means you can anticipate your agent\u2019s performance on the Test problems relatively well: each Test problem uses a near-identical principle to the corresponding Basic problem. In the past, agents have averaged getting 85% as many Test problems right as Basic problems, so there\u2019s a pretty good correlation there if<\/i> you\u2019re using a robust, general method.<\/p>\n You are provided with the Basic<\/a> and Challenge <\/a>problems to use in designing your agent. The Test and Raven\u2019s problems are hidden and will only be used when grading your project. This is to test your agents for generality: it isn\u2019t hard to design an agent that can answer questions it has already seen, just as it would not be hard to score well on a test you have already taken before. However, performing well on problems you and your agent haven\u2019t<\/i> seen before is a more reliable test of intelligence. Your grade is based solely on your agent\u2019s performance on the Basic and Test problems.<\/p>\n All problems are contained within the Problems folder of the downloadable. Problems are divided into sets, and then into individual problems. Each problem\u2019s folder has three things:<\/p>\n You should not attempt to access ProblemData.txt directly; its filename will be changed when we grade projects. Generally, you need not worry about this directory structure; all problem data will be loaded into the RavensProblem object passed to your agent\u2019s Solve method, and the filenames for the different visual representations will be included in their corresponding RavensFigures.<\/p>\n The framework code is available here<\/a> as Project-Code-Java.zip or Project-Code-Python.zip. \u00a0Both are also available from git via \u2018git clone –recursive https:\/\/github.gatech.edu\/Dilab\/KBAI-package-java.git<\/a>\u2019 (Java) or \u2018git clone –recursive https:\/\/github.gatech.edu\/Dilab\/KBAI-package-python.git<\/a>\u2019 respectively. \u00a0Note that these code packages contain a file instructing your agent to only look at those problems relevant for Project 1; updated versions of this file will be supplied for Project 2 and Project 3.<\/p>\n The downloadable package has a number of either Java or Python files: RavensProject, ProblemSet, RavensProblem, RavensFigure, RavensObject, and Agent. Of these, you should only modify the Agent class. You may make changes to the other classes to test your agent, write debug statements, etc. However, when we test your code, we will use the original versions of these files as downloaded here. Do not rely on changes to any class except for Agent to run your code. In addition to Agent, you may also write your own additional files and classes for inclusion in your project.<\/p>\n In Agent, you will find two methods: a constructor and a Solve method. The constructor will be called at the beginning of the program, so you may use this method to initialize any information necessary before your agent begins solving problems. After that, Solve will be called on each problem. You should write the Solve method to return its answer to the given question:<\/p>\n You may do all the processing within Solve, or you may write other methods and classes to help your agent solve the problems.<\/p>\n When running, the program will load questions from the Problems folder. It will then ask your agent to solve each problem one by one and write the results to ProblemResults.csv. You may check ProblemResults.csv to see how well your agent performed. You may also check SetResults.csv to view a summary of your agent\u2019s performance at the set level.<\/p>\n Included in the downloadable is the documentation for interacting with the code (API\/index.html in the downloadable). You may use this and the in-line comments to understand the structure of the problems. Briefly, however:<\/p>\n The documentation is ultimately somewhat straightforward, but it can be complicated when you\u2019re initially getting used to it. The most important things to remember are:<\/p>\n No<\/b> external libraries are permitted in Java. In Python, the only<\/b> permitted libraries are the latest version of the Python image processing library Pillow and the latest version of numpy. For installation instructions on Pillow, see this page<\/a>. For installation instructions on numpy, see this page<\/a>. No other libraries are permitted.<\/p>\n Generally, we do not allow external libraries. For Java, you may use anything contained within the default Java 8 installation. Java 8 has plenty of image processing options. We recommend using BufferedImage, and we have included a bit of sample code below for loading images into BufferedImage. If you have other suggestions, please bring them up on Piazza!<\/p>\n Python has no native support for image processing, so an external library must be used. The only<\/b> external library we support for image processing for Python is Pillow. You can install pillow simply by running easy_install pillow. More comprehensive information on installing Pillow can be found here<\/a>. We have included a code segment below on loading an image from a file with Pillow.<\/p>\n This class uses a server-side autograder to evaluate your submission. This means you can see how your code is performing against the Test problems even without seeing the problems themselves. Note that we will<\/strong> look at your submission history when evaluating your implementation: we want to see that you modified your agent over time, either to try to improve its performance or to experiment with different approaches. So, you should submit early and often. However, you are limited to 10 submissions per project<\/strong> to avoid over-stressing the autograder and to disincentivize “brute force” methods for solving the problems.<\/p>\n Below are the step-by-step submission instructions for submitting your agent to the autograder.<\/p>\n First, make sure your agent is built using the official framework. Follow the instructions under Working with the Code.<\/p>\n Then, install a couple of packages required for running the submit script by running:<\/p>\n If you are using Python 3, please update the language.txt file to say \u2018python3\u2019 instead of \u2018python\u2019.<\/p>\n In your project directory (for either language) you will find a Python script called submit.py.\u00a0 When running it, two parameters must be provided: provider and assignment. provider should always be gt: `–provider gt`. For assignment, the available parameters are P1, P2, P3, error-check, error-check-2, and error-check-3. P1, P2, and P3 submit your code against the corresponding project. error-check, error-check-2 (for P2) and error-check-3 (for P3) submit your code just for a preemptive check to make sure it is properly formatted and runnable by the autograder. We strongly recommend you first run the error check:<\/p>\n Note that this does not count as a project submission and will not be graded. It only ensures that your code is properly formatted and ready for grading prior to real submission so that you don’t waste a submission on broken code.<\/p>\n To submit project 1 for credit, run:<\/p>\n Note that running this command will count against your submission limit for P1.<\/p>\n If you have created additional files, make sure to include them with the –files parameter. For example, if you are submitting project 1 built in Java and your agent uses two extra classes called RavensTransform and RavensNetwork, you would run:<\/p>\n Note that it is not necessary to explicitly list all the files included in the project framework.<\/p>\n The script will prompt you for your login information. \u00a0Make sure to use your Georgia Tech login credentials so that we can credit you for your submission! \u00a0Optionally, you can save your login information when the script prompts you to \u201csave jwt\u201d — this will avoid the need to re-enter it on future runs.<\/p>\n Note: if you are using two-factor authentication, an additional step is required to allow the submit script to log in on your behalf — please see the instructions at https:\/\/bonnie.udacity.com\/auth_tokens\/two_factor<\/a>.<\/p>\n The script will run for a while and then return a set of results which looks something like this:<\/p>\n It will also display some summary information per problem set in the console. You may find it convenient to open the CSV-formatted results in a spreadsheet program like Excel for easy browsing and reference. The autograder will not provide you your code’s own output or access to any files your code generates in order to prevent leaking information.<\/p>\n You can check the number of submissions you have remaining (as well as some other fun statistics about your agent) at https:\/\/bonnie.udacity.com\/<\/a>.<\/p>\n If your submission errors out, you\u2019ll see a result indicating what kind of error the code encountered. \u00a0(For Java, this will be either a build or an execution error; since Python is an interpreted language, only execution errors are possible.)<\/p>\n Here are some debugging steps you can try:<\/p>\n If you encounter any problems at all when running the autograder, please post to Piazza in the autograder folder. \u00a0This helps make sure we\u2019ll see your issue and respond promptly!<\/p>\n Goel, A. (2015). Geometry, Drawings, Visual Thinking, and Imagery: Towards a Visual Turing Test of Machine Intelligence<\/a>. In Proceedings of the 29th Association for the Advancement of Artificial Intelligence Conference Workshop on Beyond the Turing Test.<\/i> Austin, Texas.<\/p>\n McGreggor, K., & Goel, A. (2014). Confident Reasoning on Raven\u2019s Progressive Matrices Tests<\/a>. In Proceedings of the 28th Association for the Advancement of Artificial Intelligence Conference. <\/i>Qu\u00e9bec City, Qu\u00e9bec.<\/p>\n Kunda, M. (2013). Visual problem solving in autism, psychometrics, and AI: the case of the Raven’s Progressive Matrices intelligence test<\/a>. Doctoral dissertation.<\/p>\n Emruli, B., Gayler, R. W., & Sandin, F. (2013). Analogical mapping and inference with binary spatter codes and sparse distributed memory<\/a>. In Neural Networks (IJCNN), The 2013 International Joint Conference on<\/i>. IEEE.<\/p>\n Little, D., Lewandowsky, S., & Griffiths, T. (2012). A Bayesian model of rule induction in Raven\u2019s progressive matrices<\/a>. In Proceedings of the 34th Annual Conference of the Cognitive Science Society<\/i>. Sapporo, Japan.<\/p>\n Kunda, M., McGreggor, K., & Goel, A. K. (2012). Reasoning on the Raven\u2019s advanced progressive matrices test with iconic visual representations<\/a>. In 34th Annual Conference of the Cognitive Science Society<\/i>. Sapporo, Japan.<\/p>\n Lovett, A., & Forbus, K. (2012). Modeling multiple strategies for solving geometric analogy problems<\/a>. In 34th Annual Conference of the Cognitive Science Society<\/i>. Sapporo, Japan.<\/p>\n Schwering, A., Gust, H., K\u00fchnberger, K. U., & Krumnack, U. (2009). Solving geometric proportional analogies with the analogy model HDTP<\/a>. In 31st Annual Conference of the Cognitive Science Society<\/i>. Amsterdam, Netherlands.<\/p>\n Joyner, D., Bedwell, D., Graham, C., Lemmon, W., Martinez, O., & Goel, A. (2015). Using Human Computation to Acquire Novel Methods for Addressing Visual Analogy Problems on Intelligence Tests. In Proceedings of the Sixth International Conference on Computational Creativity<\/i>. Provo, Utah.<\/p>\nBackground and Goals<\/h2>\n
Learning Goals<\/h3>\n
About the Test<\/h3>\n
Verbal vs. Visual<\/h3>\n
Details & Deliverables<\/h2>\n
Project Progression<\/h3>\n
\n
Grading<\/h3>\n
Performance<\/h4>\n
Revise & Reflect<\/h4>\n
Journal<\/h4>\n
Top Performers<\/h4>\n
Getting Started<\/h2>\n
\n
The Problem Sets<\/h3>\n
\n
![\"\"](\"http:\/\/lucylabs.gatech.edu\/kbai\/wp-content\/uploads\/2018\/08\/ravens.png\")
<\/p>\nThe Problems<\/h3>\n
\n
Working with the Code<\/h3>\n
The Code<\/h4>\n
\n
The Documentation<\/h4>\n
\n
\n
\n
Libraries<\/h4>\n
Image Processing<\/h4>\n
Submitting Your Code<\/h2>\n
Getting Started with the Autograder<\/h4>\n
pip install requests future<\/pre>\n
python submit.py --provider gt --assignment error-check<\/pre>\n
Submitting a Project<\/h4>\n
python submit.py --provider gt --assignment P1<\/pre>\n
python submit.py --provider gt --assignment P1 --files RavensTransform.java RavensNetwork.java<\/pre>\n
Logging In<\/h4>\n
Getting Your Results<\/h4>\n
Problem,Correct?,Correct Answer,Agent's Answer\r\n\"Challenge Problem B-04\",0,4,1\r\n\"Basic Problem B-12\",1,1,1\r\n...<\/pre>\n
Submission Errors<\/h4>\n
\n
\n
Relevant Resources<\/h1>\n