[et_pb_section fb_built=”1″ admin_label=”Section” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_row admin_label=”Project Title” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.16″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” header_font=”|700||on|||||” global_colors_info=”{}”]<\/p>\n
[\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row use_custom_gutter=”on” gutter_width=”1″ admin_label=”row” _builder_version=”4.16″ _module_preset=”default” background_size=”initial” background_position=”top_left” background_repeat=”repeat” width=”100%” custom_padding=”0px||0px||false|false” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_divider color=”#eeeeee” divider_position=”center” divider_weight=”3px” _builder_version=”4.16″ _module_preset=”default” width=”25%” custom_padding=”30px||30px||true|false” global_colors_info=”{}”][\/et_pb_divider][et_pb_blurb title=”Table of Contents” use_icon=”on” font_icon=”h||divi||400″ icon_color=”rgba(0,0,0,0.05)” icon_placement=”left” image_icon_width=”100px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_level=”h2″ header_font_size=”26px” height=”38px” icon_font_size=”100px” global_colors_info=”{}”][\/et_pb_blurb][et_pb_blurb title=”Overview” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#overview” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”About the Project” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#about” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Your Implementation” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#implementation” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Contents of Report” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#report” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Testing Recommendations” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#testing” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Submission Requirements” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#submission” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Grading Information” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#grading” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Development Guidelines” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#guidelines” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_blurb title=”Optional Resources” use_icon=”on” font_icon=”$||divi||400″ icon_color=”#000000″ image_icon_background_color=”#FFFFFF” icon_placement=”left” image_icon_width=”16px” content_max_width=”100%” _builder_version=”4.16″ _module_preset=”default” header_font_size=”16px” header_line_height=”2em” image_icon_custom_padding=”8px|8px|8px|8px|false|false” custom_margin=”|||118px|false|false” custom_padding=”10px|||0px|false|false” link_option_url=”#optional” border_radii_image=”on|100%|100%|100%|100%” border_width_all_image=”2px” border_color_all_image=”#000000″ icon_font_size=”16px” use_circle=”on” use_circle_border=”on” circle_border_color=”#b856c7″ circle_color=”#FFFFFF” global_colors_info=”{}” font_icon__hover_enabled=”on|hover” font_icon__hover=”%22||divi||400″ custom_padding__hover=”|||10px|false|false” custom_padding__hover_enabled=”on|hover” image_icon_background_color__sticky_enabled=”#7EBEC5″ image_icon_background_color__sticky=”#7EBEC5″][\/et_pb_blurb][et_pb_divider color=”#eeeeee” divider_position=”center” divider_weight=”3px” _builder_version=”4.16″ _module_preset=”default” width=”25%” custom_padding=”30px||30px||true|false” global_colors_info=”{}”][\/et_pb_divider][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”Revisions” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text admin_label=”Text” _builder_version=”4.17.4″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n
This assignment is subject to change up until 3 weeks prior to the due date. We do not anticipate changes; any changes will be logged in this section.<\/span><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”1 Overview” module_id=”overview” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.16″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n In this assignment, you <\/span>implement a <\/span>Reinforcement Learning algorithm called Q-learning<\/span>, which is a model-free RL algorithm. You will also ex<\/span>t<\/span>end your Q-learner implementation by adding a Dyna, model-based, <\/span>component. <\/span>You will submit the code for the project in <\/span>Gradescope<\/span> SUBMISSION. <\/span>There is no report associated with this assignment.<\/span><\/span>\u00a0<\/span><\/p>\n [\/et_pb_text][et_pb_text admin_label=”1.1 Learning Objectives” _builder_version=”4.16″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n The specific learning objectives for this assignment are focused on the following areas:<\/span>\u00a0<\/span><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”2 About The Project” module_id=”about” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.16″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n I<\/span>n this project, you will implement the Q-Learning and Dyna-Q solutions to the reinforcement learning problem. You will apply them to a navigation problem in this project. In a later project, you will apply them to trading. The reason for working with the navigation problem first is that, as you will see, navigation is an easy problem to work with and understand. Note that your Q-Learning code really shouldn\u2019t care which problem it is solving. The difference is that you need to wrap the learner in a different code that frames the problem for the learner as necessary.<\/span><\/span>\u00a0<\/span><\/p>\n [\/et_pb_text][\/et_pb_column][\/et_pb_row][et_pb_row admin_label=”3 Your Implementation” module_id=”implementation” _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”][et_pb_text _builder_version=”4.16″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n For this project, we have created testqlearner.py that automates the testing of your Q-Learner in the navigation problem.<\/span>\u00a0<\/span><\/p>\n Overall, your tasks for this project include:<\/span>\u00a0<\/span><\/p>\n You must write your own code for this project. You are NOT allowed to use other people\u2019s code or packages to implement the Q-learner. For this assignment, we will test only your code (there is no report component).<\/span>\u00a0<\/span><\/p>\n Before the deadline, make sure to pre-validate your submission using Gradescope TESTING. Once you are satisfied with the results in testing, submit the code to Gradescope SUBMISSION. <\/span>Only code submitted to Gradescope SUBMISSION will be graded. If you submit your code to Gradescope TESTING and have not also submitted your code to Gradescope SUBMISSION, you will receive a zero (0).<\/span><\/b>\u00a0<\/span><\/p>\n [\/et_pb_text][et_pb_text admin_label=”3.1 Getting Started” _builder_version=”4.18.0″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n To make it easier to get started on the project and focus on the concepts involved, you will be given a starter framework. This framework assumes you have already set up the <\/span><\/span>local environment<\/span><\/span><\/a> and <\/span><\/span>ML4T Software<\/span><\/span><\/a>. <\/span>The framework for Project <\/span>7<\/span> can be obtained from:\u202f<\/span><\/span>QLearning_Robot_2022Fall.zip<\/a>.\u202f<\/span><\/span>\u00a0<\/span><\/p>\n [\/et_pb_text][et_pb_image src=”http:\/\/lucylabs.gatech.edu\/ml4t\/wp-content\/uploads\/2021\/09\/Screen-Shot-2021-09-26-at-12.15.06-AM.png” title_text=”Screen Shot 2021-09-26 at 12.15.06 AM” admin_label=”Image” _builder_version=”4.16″ _module_preset=”f2b74940-54e3-4aec-b2e5-e58531b6bfc5″ global_colors_info=”{}”][\/et_pb_image][et_pb_text admin_label=”3.1 Cont.” _builder_version=”4.18.0″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n Extract its contents into the base directory (e.g., ML4T_2022Fall). This will add a new folder called \u201cqlearning_robot\u201d to the course directory structure:\u202f\u202f<\/span>\u00a0<\/span><\/p>\n The framework for Project 7 can be obtained in the qlearning_robot folder alone. Within the qlearning_robot folder are several files:\u00a0<\/span>\u00a0<\/span><\/p>\n Note: Example navigation problems are provided in the qlearning_robot\/testworlds directory<\/span><\/i>.<\/span>\u00a0<\/span><\/p>\n Your q-learning class will be implemented in the QLearner.py file. The testqlearner.py file contains a simple testing scaffold that you can use to test your learners, which is useful for debugging. It must also be modified to run the experiments. The grade_robot_qlearners.py file is a local pre-validation script that mirrors the script used in the Gradescope TESTING environment.<\/span>\u00a0<\/span><\/p>\n [\/et_pb_text][et_pb_text admin_label=”3.2 Task & Requirements” _builder_version=”4.16″ _module_preset=”d93d428f-8726-40a1-9b24-ea6370f64b6f” global_colors_info=”{}”]<\/p>\n You will implement the following files:\u00a0\u00a0<\/span>\u00a0<\/span><\/p>\n All your learner code must be placed into the file above. No other code files will be accepted. The testqlearner.py file that is used to conduct your experiments must also reside in the qlearning_robot and is run using the following command:<\/span>\u00a0<\/span><\/p>\n [\/et_pb_text][et_pb_code _builder_version=”4.16″ _module_preset=”default” global_colors_info=”{}”]1 Overview<\/h2>\n
1.1 Learning Objectives<\/h3>\n
\n
2 About The Project<\/h2>\n
3 Your Implementation<\/h2>\n
\n
<\/ul>\n
3.1 Getting Started<\/h3>\n
\n
3.2 Task & Requirements<\/h3>\n
\n