General Self-Motivation and Strategy Identification: Case Studies Based on Sokoban and Pac-Man
In this paper, we use empowerment, a recently introduced biologically inspired measure, to allow an AI player to assign utility values to potential future states within a previously unencountered game without requiring explicit specification of goal states. We further introduce strategic affinity, a...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on computational intelligence and AI in games. Jg. 6; H. 1; S. 1 - 17 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
01.03.2014
|
| Schlagworte: | |
| ISSN: | 1943-068X, 1943-0698 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | In this paper, we use empowerment, a recently introduced biologically inspired measure, to allow an AI player to assign utility values to potential future states within a previously unencountered game without requiring explicit specification of goal states. We further introduce strategic affinity, a method of grouping action sequences together to form "strategies," by examining the overlap in the sets of potential future states following each such action sequence. We also demonstrate an information-theoretic method of predicting future utility. Combining these methods, we extend empowerment to soft-horizon empowerment which enables the player to select a repertoire of action sequences that aim to maintain anticipated utility. We show how this method provides a proto-heuristic for nonterminal states prior to specifying concrete game goals, and propose it as a principled candidate model for "intuitive" strategy selection, in line with other recent work on "self-motivated agent behavior." We demonstrate that the technique, despite being generically defined independently of scenario, performs quite well in relatively disparate scenarios, such as a Sokoban-inspired box-pushing scenario and in a Pac-Man-inspired predator game, suggesting novel and principle-based candidate routes toward more general game-playing algorithms. |
|---|---|
| AbstractList | In this paper, we use empowerment, a recently introduced biologically inspired measure, to allow an AI player to assign utility values to potential future states within a previously unencountered game without requiring explicit specification of goal states. We further introduce strategic affinity, a method of grouping action sequences together to form "strategies," by examining the overlap in the sets of potential future states following each such action sequence. We also demonstrate an information-theoretic method of predicting future utility. Combining these methods, we extend empowerment to soft-horizon empowerment which enables the player to select a repertoire of action sequences that aim to maintain anticipated utility. We show how this method provides a proto-heuristic for nonterminal states prior to specifying concrete game goals, and propose it as a principled candidate model for "intuitive" strategy selection, in line with other recent work on "self-motivated agent behavior." We demonstrate that the technique, despite being generically defined independently of scenario, performs quite well in relatively disparate scenarios, such as a Sokoban-inspired box-pushing scenario and in a Pac-Man-inspired predator game, suggesting novel and principle-based candidate routes toward more general game-playing algorithms. |
| Author | Polani, Daniel Nehaniv, Chrystopher L. Anthony, Tom |
| Author_xml | – sequence: 1 givenname: Tom surname: Anthony fullname: Anthony, Tom email: research@tomanthony.co.uk organization: Adaptive Syst. Res. Group, Univ. of Hertfordshire, Hatfield, UK – sequence: 2 givenname: Daniel surname: Polani fullname: Polani, Daniel email: D.Polani@herts.ac.uk organization: Adaptive Syst. Res. Group, Univ. of Hertfordshire, Hatfield, UK – sequence: 3 givenname: Chrystopher L. surname: Nehaniv fullname: Nehaniv, Chrystopher L. email: haniv@herts.ac.uk organization: Adaptive Syst. Res. Group, Univ. of Hertfordshire, Hatfield, UK |
| BookMark | eNqFkE9Lw0AQxRdRsNZ-gl5y9JK6f5JN1lsNWgMtCq3gyTDZTGQ13dRsKvTbmzalBy_OZd4w7zcw74qc29oiIWNGJ4xRdbtK0mk6m3DKxIRzFYqIn5EBU4HwqVTx-UnHb5dk5Nwn7UoIIbkckPcZWmyg8pZYlf6ibs0PtKa2HtjCW7YNtPix89ICbWtKow-7Oy8Bh912Wxh03n03FF6HLOuvOocefQHtL8Bek4sSKoejYx-S18eHVfLkz59naTKd-1oo2foy1iGHkubAyyinNBCdokrQMseIB0BBlxqKXBUyDhQyxaRgyEGHjNKCB2JIbvq7m6b-3qJrs7VxGqsKLNZblzEZsVCpIJadVfVW3dTONVhm2rSHv7pvTZUxmu1jzfpYs32s2THWjhV_2E1j1tDs_qHGPWUQ8URIGUecKfELvxCFjw |
| CODEN | TCIARR |
| CitedBy_id | crossref_primary_10_1109_TG_2017_2737145 crossref_primary_10_20965_jaciii_2015_p0867 crossref_primary_10_1088_2632_072X_adf2ec crossref_primary_10_1145_3404197 crossref_primary_10_3390_e16052789 crossref_primary_10_3389_frobt_2017_00025 crossref_primary_10_3390_e16063357 crossref_primary_10_1093_logcom_exu058 |
| Cites_doi | 10.1016/S0004-3702(01)00109-6 10.1007/BF01448847 10.1613/jair.3125 10.1109/TEVC.2006.890271 10.1007/11840541_46 10.1016/S0925-7721(99)00017-6 10.1080/net.12.3.241.253 10.1007/978-3-642-21314-4_37 10.1088/0954-898X/3/2/009 10.1142/S0219525912500798 10.1007/978-3-540-27833-7_17 10.7551/mitpress/3115.003.0030 10.1007/11553090_75 10.1109/TIT.1953.1188565 10.1371/journal.pone.0004018 10.1109/EH.2004.1310828 10.1109/TIT.1972.1054855 10.2466/pr0.1977.41.1.3 10.1140/epjb/e2008-00175-0 10.1109/CIG.2009.5286469 10.1109/TAMD.2010.2056368 10.1177/1059712310392389 10.1147/rd.116.0601 10.1038/236 10.1162/089976601753195969 10.1037/h0054663 10.1162/EVCO_a_00025 10.7551/mitpress/3585.001.0001 10.1145/307400.307435 10.1007/3-540-48304-7_45 10.1109/CEC.2005.1554676 10.1038/nature05464 10.1086/227496 10.1002/j.1538-7305.1948.tb01338.x 10.1007/978-3-540-74913-4_38 |
| ContentType | Journal Article |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TCIAIG.2013.2295372 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE/IET Electronic Library CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Computer and Information Systems Abstracts |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISSN | 1943-0698 |
| EndPage | 17 |
| ExternalDocumentID | 10_1109_TCIAIG_2013_2295372 6687219 |
| Genre | orig-research |
| GroupedDBID | 0R~ 29F 4.4 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABJNI ABQJQ ABVLG ACIWK AENEX AETIX AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD HZ~ IEDLZ IFIPE IPLJI JAVBF M43 O9- OCL P2P RIA RIE RNS AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c396t-68c52af0ba2f7b0043ba20930fbe724a0acfcadb9d6849e191631e2ac5100d243 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 15 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000333115100001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1943-068X |
| IngestDate | Sun Sep 28 01:13:36 EDT 2025 Sat Nov 29 03:28:48 EST 2025 Tue Nov 18 22:41:41 EST 2025 Tue Aug 26 16:49:30 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 1 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c396t-68c52af0ba2f7b0043ba20930fbe724a0acfcadb9d6849e191631e2ac5100d243 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| PQID | 1671599486 |
| PQPubID | 23500 |
| PageCount | 17 |
| ParticipantIDs | ieee_primary_6687219 proquest_miscellaneous_1671599486 crossref_primary_10_1109_TCIAIG_2013_2295372 crossref_citationtrail_10_1109_TCIAIG_2013_2295372 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-03-01 |
| PublicationDateYYYYMMDD | 2014-03-01 |
| PublicationDate_xml | – month: 03 year: 2014 text: 2014-03-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationTitle | IEEE transactions on computational intelligence and AI in games. |
| PublicationTitleAbbrev | TCIAIG |
| PublicationYear | 2014 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| References | ref57 ref12 ref14 ref52 ref54 ref10 ref16 ref19 ref18 shannon (ref3) 1950; 41 singh (ref41) 2005 anthony (ref34) 2008 ref51 ref50 syed (ref6) 2003; 26 prokopenko (ref20) 2006; 4095 klyubin (ref24) 2005; 3630 oudeyer (ref43) 2008 ref47 ref42 simon (ref11) 1957 berger (ref44) 2003; 53 singh (ref33) 2004 varela (ref9) 1992 pfeifer (ref8) 2006 ref4 veness (ref40) 2011; 40 tishby (ref48) 1999 m ller (ref55) 2010 genesereth (ref46) 2005; 26 ref35 tisdell (ref13) 1996 ref30 ref32 schmidhuber (ref53) 2008 chaslot (ref7) 2008 ref2 ref1 ref39 ref38 barlow (ref17) 1959 pearl (ref45) 1984 tishby (ref15) 2010 schmidhuber (ref36) 1991 pearl (ref31) 2000 ref23 ref26 ref25 ref22 ref21 m ller (ref56) 2010 ref28 slonim (ref49) 2003 ref27 ref29 steels (ref37) 2004; 3139 mccarthy (ref5) 2007 |
| References_xml | – ident: ref50 doi: 10.1016/S0004-3702(01)00109-6 – volume: 41 start-page: 256 year: 1950 ident: ref3 article-title: Programming a computer for playing chess publication-title: Philosoph Mag – ident: ref2 doi: 10.1007/BF01448847 – volume: 40 start-page: 95 year: 2011 ident: ref40 article-title: A Monte-Carlo AIXI approximation publication-title: J Artif Intell Res doi: 10.1613/jair.3125 – ident: ref38 doi: 10.1109/TEVC.2006.890271 – volume: 4095 start-page: 558 year: 2006 ident: ref20 publication-title: From Animals to Animats 9 doi: 10.1007/11840541_46 – ident: ref51 doi: 10.1016/S0925-7721(99)00017-6 – ident: ref18 doi: 10.1080/net.12.3.241.253 – ident: ref30 doi: 10.1007/978-3-642-21314-4_37 – ident: ref19 doi: 10.1088/0954-898X/3/2/009 – ident: ref57 doi: 10.1142/S0219525912500798 – volume: 3139 start-page: 231 year: 2004 ident: ref37 publication-title: Embodied Artificial Intelligence doi: 10.1007/978-3-540-27833-7_17 – start-page: 222 year: 1991 ident: ref36 publication-title: From Animals to Animats doi: 10.7551/mitpress/3115.003.0030 – year: 2010 ident: ref56 publication-title: ?Fuego-GB prototype at the Human Machine Competition in Barcelona 2010 A tournament report and analysis – volume: 3630 start-page: 744 year: 2005 ident: ref24 publication-title: Advances in Artificial Life doi: 10.1007/11553090_75 – ident: ref25 doi: 10.1109/TIT.1953.1188565 – ident: ref26 doi: 10.1371/journal.pone.0004018 – volume: 26 start-page: 138 year: 2003 ident: ref6 article-title: Arimaa?A new game designed to be difficult for computers publication-title: Int Comput Games Assoc J – start-page: 93 year: 2008 ident: ref43 article-title: How can we define intrinsic motivation? publication-title: Proc 6th Int Conf Epigenetic Robot Modeling Cogn Develop Robot Syst – year: 1996 ident: ref13 publication-title: Bounded Rationality and Economic Evolution A Contribution to Decision Making Economics and Management – ident: ref29 doi: 10.1109/EH.2004.1310828 – year: 2010 ident: ref55 publication-title: ?Challenges in Monte-Carlo tree search ? – ident: ref28 doi: 10.1109/TIT.1972.1054855 – year: 2007 ident: ref5 publication-title: ?What is artificial intelligence?? – ident: ref1 doi: 10.2466/pr0.1977.41.1.3 – year: 1957 ident: ref11 publication-title: Models of Man Social and Rational – ident: ref22 doi: 10.1140/epjb/e2008-00175-0 – start-page: 368 year: 1999 ident: ref48 article-title: The information bottleneck method publication-title: Proc 37th Annu Allerton Conf Commun Control Comput – ident: ref52 doi: 10.1109/CIG.2009.5286469 – start-page: 1281 year: 2005 ident: ref41 article-title: Intrinsically motivated reinforcement learning publication-title: Proc 18th Annu Conf Neural Inf Process Syst – year: 1992 ident: ref9 publication-title: The Embodied Mind Cognitive Science and Human Experience – ident: ref39 doi: 10.1109/TAMD.2010.2056368 – year: 2003 ident: ref49 publication-title: The Information Bottleneck Theory and Applications – ident: ref35 doi: 10.1177/1059712310392389 – ident: ref4 doi: 10.1147/rd.116.0601 – year: 2008 ident: ref53 publication-title: ?Driven by compression progress A simple principle explains essential aspects of subjective beauty novelty surprise interestingness attention curiosity creativity art science music jokes ? – ident: ref47 doi: 10.1038/236 – start-page: 512 year: 2004 ident: ref33 article-title: Predictive state representations: A new theory for modeling dynamical systems publication-title: Proc 20th Conf Uncertainty Artif Intell – year: 1984 ident: ref45 publication-title: Heuristics Intelligent Search Strategies for Computer Problem Solving – ident: ref21 doi: 10.1162/089976601753195969 – ident: ref16 doi: 10.1037/h0054663 – ident: ref54 doi: 10.1162/EVCO_a_00025 – year: 2006 ident: ref8 publication-title: How the Body Shapes the Way We Think A New View of Intelligence doi: 10.7551/mitpress/3585.001.0001 – ident: ref14 doi: 10.1145/307400.307435 – year: 2000 ident: ref31 publication-title: Causality Models Reasoning and Inference – ident: ref10 doi: 10.1007/3-540-48304-7_45 – ident: ref23 doi: 10.1109/CEC.2005.1554676 – ident: ref42 doi: 10.1038/nature05464 – ident: ref12 doi: 10.1086/227496 – volume: 26 start-page: 62 year: 2005 ident: ref46 article-title: General game playing: Overview of the AAAI competition publication-title: AI Mag – ident: ref27 doi: 10.1002/j.1538-7305.1948.tb01338.x – start-page: 216 year: 2008 ident: ref7 article-title: Monte-Carlo tree search: A new framework for game AI publication-title: Proc 4th Artif Intell Interactive Digit Entertain Conf – start-page: 25 year: 2008 ident: ref34 publication-title: Artificial Life XI – volume: 53 start-page: 1 year: 2003 ident: ref44 article-title: Living information theory publication-title: IEEE Inf Theory Soc Newslett – year: 2010 ident: ref15 publication-title: Perception-Reason-Action Cycle Models Algorithms and Systems – ident: ref32 doi: 10.1007/978-3-540-74913-4_38 – start-page: 217 year: 1959 ident: ref17 publication-title: Sensory Communication |
| SSID | ssj0000333626 |
| Score | 2.077213 |
| Snippet | In this paper, we use empowerment, a recently introduced biologically inspired measure, to allow an AI player to assign utility values to potential future... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 1 |
| SubjectTerms | Affinity Artificial intelligence Artificial intelligence (AI) Cognition Computers Entropy Games information theory Intelligence Mutual information Players Random variables Specifications Strategy Utilities |
| Title | General Self-Motivation and Strategy Identification: Case Studies Based on Sokoban and Pac-Man |
| URI | https://ieeexplore.ieee.org/document/6687219 https://www.proquest.com/docview/1671599486 |
| Volume | 6 |
| WOSCitedRecordID | wos000333115100001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1943-0698 dateEnd: 20171231 omitProxy: false ssIdentifier: ssj0000333626 issn: 1943-068X databaseCode: RIE dateStart: 20090101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD7M4YMvTp3ivBHBx3VLm5o2vs3hdODGYFP2ZMlyAXG0sovgvzdps4Iogm8p5JDSryfnJOfyAVxhKogNZxrPTXJLYSY8JhX2JPa5JsYkFnXcz4_RcBhPp2xUgWZZC6OUypPPVMsO81i-zMTaXpW1KY3NgYVtwVYU0aJWq7xPwYTYzip5EDm0hAXx1DUZ8jFrT7r9Tv_eZnKRliWwJlHwzRDlzCo_tuPcxvRq_3u7Pdh1viTqFODvQ0WlB1Db8DQgp7Z1eHG9pdFYzbU3KBnNEE8lcu1pP1FRsqvdHd4N6hr7hlyaIbo1DxIZkXH2ZnaAQnTEhTfg6SE89e4m3QfP8Sp4gjC68mgsrgOu8YwHOrJqS8wIM4L1TEVByDEXWnA5Y5LGIVPmREeJrwIujP5iGYTkCKpplqpjQCoSikhhvBbbGI6ImEsd-tyXMypCs0wDgs1HToRrOm65L-ZJfvjALCmQSSwyiUOmAc1S6L3oufH39LoFo5zqcGjA5QbNxKiMjYPwVGXrZeLTyDhxLIzpye-ip7BjFgiLVLMzqK4Wa3UO2-Jj9bpcXOT_3RdpGNPy |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD7oFPTFecV5jeCj1bSJaePbHM4NtyE4ZU-WNElBHK24TfDfm7RZQRTBtxRySOnXk3OSc_kATjGTxIYzjeemhKUwkx5XGnsK-yIlxiSWddxPvXAwiEYjfr8AZ1UtjNa6SD7T53ZYxPJVLmf2quyCscgcWPgiLF1SGuCyWqu6UcGE2N4qRRiZWsqCaOTaDPmYXwxb3Wb31uZykXNLYU3C4JspKrhVfmzIhZVp1__3fuuw5rxJ1Czh34AFnW1Cfc7UgJzibsGz6y6NHvQ49foVpxkSmUKuQe0nKot2U3eLd4VaxsIhl2iIrs2DQkbkIX81e0Apei-k1xfZNjy2b4atjueYFTxJOJt6LJKXgUhxIoI0tIpLzAhzgtNEhwEVWMhUCpVwxSLKtTnTMeLrQEijwVgFlOxALcszvQtIh1ITJY3fYlvDERkJlVJf-CphkpplGhDMP3IsXdtxy34xjovjB-ZxiUxskYkdMg04q4Teyq4bf0_fsmBUUx0ODTiZoxkbpbGREJHpfDaJfRYaN47TiO39LnoMK51hvxf3uoO7fVg1i9Ey8ewAatP3mT6EZfkxfZm8HxX_4BeX7dc5 |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=General+Self-Motivation+and+Strategy+Identification%3A+Case+Studies+Based+on+Sokoban+and+Pac-Man&rft.jtitle=IEEE+transactions+on+computational+intelligence+and+AI+in+games.&rft.au=Anthony%2C+Tom&rft.au=Polani%2C+Daniel&rft.au=Nehaniv%2C+Chrystopher+L&rft.date=2014-03-01&rft.issn=1943-068X&rft.eissn=1943-0698&rft.volume=6&rft.issue=1&rft.spage=1&rft.epage=17&rft_id=info:doi/10.1109%2FTCIAIG.2013.2295372&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1943-068X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1943-068X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1943-068X&client=summon |