Intuitive robot programming through environment perception, augmented reality simulation and automated program verification
The increasing complexity of products and machines as well as short production cycles with small lot sizes present great challenges to production industry. Both, the programming of industrial robots in online mode using hand-held control devices or in offline mode using text-based programming requir...
Gespeichert in:
| Veröffentlicht in: | Procedia CIRP Jg. 76; S. 161 - 166 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
Elsevier B.V
2018
|
| Schlagworte: | |
| ISSN: | 2212-8271, 2212-8271 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | The increasing complexity of products and machines as well as short production cycles with small lot sizes present great challenges to production industry. Both, the programming of industrial robots in online mode using hand-held control devices or in offline mode using text-based programming requires specific knowledge of robotics and manufacturer-dependent robot control systems. In particular for small and medium-sized enterprises the machine control software needs to be easy, intuitive and usable without time-consuming learning steps, even for employees with no in-depth knowledge of information technology. To simplify the programming of application programs for industrial robots, we extended a cloud-based, task-oriented robot control system with environment perception and plausibility check functions. For the environment perception a depth camera and pointcloud processing hardware were installed. We detect objects located in the robot’s workspace by pointcloud processing with ROS and the PCL and add them to the augmented reality user interface of the robot control. The combination of process knowledge from task-oriented application programming and information about available workpieces from automated image processing enables a plausibility check and verification of the robot program before execution. After a robot program has been approved by the plausibility check, it is tested in an augmented reality simulation for collisions with the detected objects before deployment to the physical robot hardware. Experiments were carried out to evaluate the effectiveness of the developed extensions and confirmed their functionality. |
|---|---|
| AbstractList | The increasing complexity of products and machines as well as short production cycles with small lot sizes present great challenges to production industry. Both, the programming of industrial robots in online mode using hand-held control devices or in offline mode using text-based programming requires specific knowledge of robotics and manufacturer-dependent robot control systems. In particular for small and medium-sized enterprises the machine control software needs to be easy, intuitive and usable without time-consuming learning steps, even for employees with no in-depth knowledge of information technology. To simplify the programming of application programs for industrial robots, we extended a cloud-based, task-oriented robot control system with environment perception and plausibility check functions. For the environment perception a depth camera and pointcloud processing hardware were installed. We detect objects located in the robot’s workspace by pointcloud processing with ROS and the PCL and add them to the augmented reality user interface of the robot control. The combination of process knowledge from task-oriented application programming and information about available workpieces from automated image processing enables a plausibility check and verification of the robot program before execution. After a robot program has been approved by the plausibility check, it is tested in an augmented reality simulation for collisions with the detected objects before deployment to the physical robot hardware. Experiments were carried out to evaluate the effectiveness of the developed extensions and confirmed their functionality. |
| Author | Wassermann, Jonas Krüger, Jörg Vick, Axel |
| Author_xml | – sequence: 1 givenname: Jonas surname: Wassermann fullname: Wassermann, Jonas email: wassermann@iat.tu-berlin.de organization: Technische Universität Berlin, Pascalstrasse 8-9, 10587 Berlin, Germany – sequence: 2 givenname: Axel surname: Vick fullname: Vick, Axel organization: Fraunhofer Institute for Production Systems and Design Technology IPK, Pascalstrae 8-9, 10587 Berlin, Germany – sequence: 3 givenname: Jörg surname: Krüger fullname: Krüger, Jörg organization: Technische Universität Berlin, Pascalstrasse 8-9, 10587 Berlin, Germany |
| BookMark | eNqFUEtLAzEQDlLBWvsPPOQH2DXJvj0IUnwUCl70HLLZ2e2UblKy2YXin3e37UE86FxmmO_BzHdNJsYaIOSWs4Azntxvg72zGl0gGM8CxgMWJhdkKgQXi0ykfPJjviLztt2yodKIhVxMydfK-A499kCdLayng1ntVNOgqanfONvVGwqmR2dNA2bAwWnYe7TmjqquHndQUgdqh_5AW2y6nRpRqkw5ELxt1Eg429IeHFaoj5QbclmpXQvzc5-Rz5fnj-XbYv3-ulo-rRc6jIVfpGEJWQxxonJIqpzzIi0TzdNYRHmiIUkYizXoVIVFziCqoijKRK5LVrFUFFyFMxKdfLWzbeugknuHjXIHyZkcM5RbecpQjhlKxuWQ4SB7-CXT6I-He6dw95_48SSG4bEewclWIxgNJTrQXpYW_zb4BrNRlk4 |
| CitedBy_id | crossref_primary_10_1016_j_ifacol_2019_11_307 crossref_primary_10_3390_robotics11050113 crossref_primary_10_1093_jcde_qwad061 crossref_primary_10_3390_act12080323 crossref_primary_10_3390_app11125592 crossref_primary_10_1007_s42452_025_06923_4 crossref_primary_10_1007_s10055_024_01021_z crossref_primary_10_1016_j_jii_2021_100294 crossref_primary_10_1088_1757_899X_521_1_012017 crossref_primary_10_1080_00207543_2020_1834640 |
| Cites_doi | 10.1109/ETFA.2017.8247749 10.1109/WCNC.2014.6953084 10.1109/INDIN.2016.7819154 10.1109/IROS.2004.1389914 10.1109/ISMAR.2003.1240739 10.1109/CW.2009.14 10.1109/ICARCV.2010.5707399 10.1109/ROBOT.2002.1013644 10.1109/RCAR.2016.7784014 10.1109/VECIMS.2007.4373930 |
| ContentType | Journal Article |
| Copyright | 2018 |
| Copyright_xml | – notice: 2018 |
| DBID | 6I. AAFTH AAYXX CITATION |
| DOI | 10.1016/j.procir.2018.01.036 |
| DatabaseName | ScienceDirect Open Access Titles Elsevier:ScienceDirect:Open Access CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 2212-8271 |
| EndPage | 166 |
| ExternalDocumentID | 10_1016_j_procir_2018_01_036 S2212827118300507 |
| GroupedDBID | 0R~ 0SF 4.4 457 6I. AACTN AAEDT AAEDW AAFTH AAIKJ AALRI AAXUO ABMAC ACGFS ADBBV ADEZE AEXQZ AFTJW AGHFR AITUG ALMA_UNASSIGNED_HOLDINGS AMRAJ EBS EJD FDB HZ~ IXB KQ8 M41 M~E NCXOZ O-L O9- OK1 RIG ROL SSZ AAYWO AAYXX ACVFH ADCNI ADVLN AEUPX AFPUW AIGII AKBMS AKRWK AKYEP CITATION |
| ID | FETCH-LOGICAL-c352t-73de85e56a9e6f911b7d6c1752496ce66005cec7a3b90e4f444829cd0f072b1a3 |
| ISICitedReferencesCount | 17 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000547342800030&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2212-8271 |
| IngestDate | Wed Nov 05 20:55:01 EST 2025 Tue Nov 18 21:51:16 EST 2025 Wed May 17 00:06:54 EDT 2023 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | Industrial Robotics Robot Programming Augmented Reality |
| Language | English |
| License | This is an open access article under the CC BY-NC-ND license. |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c352t-73de85e56a9e6f911b7d6c1752496ce66005cec7a3b90e4f444829cd0f072b1a3 |
| OpenAccessLink | https://dx.doi.org/10.1016/j.procir.2018.01.036 |
| PageCount | 6 |
| ParticipantIDs | crossref_primary_10_1016_j_procir_2018_01_036 crossref_citationtrail_10_1016_j_procir_2018_01_036 elsevier_sciencedirect_doi_10_1016_j_procir_2018_01_036 |
| PublicationCentury | 2000 |
| PublicationDate | 2018 2018-00-00 |
| PublicationDateYYYYMMDD | 2018-01-01 |
| PublicationDate_xml | – year: 2018 text: 2018 |
| PublicationDecade | 2010 |
| PublicationTitle | Procedia CIRP |
| PublicationYear | 2018 |
| Publisher | Elsevier B.V |
| Publisher_xml | – name: Elsevier B.V |
| References | R. Marin, P. J. Sanz, and J. S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), volume 3, pages 2725–2730, 2002. Z. Pan, J. Polden, N. Larkin, S. V. Duin, and J. Norrish. Recent progress on programming methods for industrial robots. In ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics), pages 1–8, June 2010. Y. Lin, S. Song, and M. Q. H. Meng. The implementation of augmented reality in a robotic teleoperation system. In 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR), pages 134–139, June 2016. S. M. Abbas, S. Hassan, and J. Yun. Augmented reality based teaching pendant for industrial robot. In 2012 12th International Conference on Control, Automation and Systems, pages 2210–2213, Oct 2012. C. L. Ng, T. C. Ng, T. A. N. Nguyen, G. Yang, and W. Chen. Intuitive robot tool path teaching using laser and camera in augmented reality environment. In 2010 11th International Conference on Control Automation Robotics Vision, pages 114–119, Dec 2010. I. Mal, D. Sedlek, and P. Leit£o. Augmented reality experiments with industrial robot in industry 4.0 environment. In 2016 IEEE 14th International Conference on Industrial Informatics (INDIN), pages 176–181, July 2016. H. Fang, S. K. Ong, and A. Y. C. Nee. Robot programming using augmented reality. In 2009 International Conference on CyberWorlds, pages 13–20, Sept 2009. R. Bischoff and A. Kazi. Perspectives on augmented reality based human-robot interaction with industrial robots. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), volume 4, pages 3226–3231 vol.4, Sept 2004. G. Reinhart, W. Vogl, and I. Kresse. A projection-based user interface for industrial robots. In 2007 IEEE Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems, pages 67– 71, June 2007. Jan Guhl, Son Tung Nguyen, and Jörg Krüger. Concept and architecture for programming industrial robots using augmented reality with mobile devices like microsoft hololens. In Emerging Technologies And Factory Automation, USA, 2017. IEEE. T. Pettersen, J. Pretlove, C. Skourup, T. Engedal, and T. Lokstad. Augmented reality for programming industrial robots. In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings., pages 319–320, Oct 2003. P. H. Chiu, P. H. Tseng, and K. T. Feng. Cloud computing based mobile augmented reality interactive system. In 2014 IEEE Wireless Communications and Networking Conference (WCNC), pages 3320–3325, April 2014. 10.1016/j.procir.2018.01.036_bib00010 10.1016/j.procir.2018.01.036_bib00012 10.1016/j.procir.2018.01.036_bib0001 10.1016/j.procir.2018.01.036_bib00011 10.1016/j.procir.2018.01.036_bib0002 10.1016/j.procir.2018.01.036_bib0003 10.1016/j.procir.2018.01.036_bib0004 10.1016/j.procir.2018.01.036_bib0005 10.1016/j.procir.2018.01.036_bib0006 10.1016/j.procir.2018.01.036_bib0007 10.1016/j.procir.2018.01.036_bib0008 10.1016/j.procir.2018.01.036_bib0009 |
| References_xml | – reference: R. Marin, P. J. Sanz, and J. S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292), volume 3, pages 2725–2730, 2002. – reference: H. Fang, S. K. Ong, and A. Y. C. Nee. Robot programming using augmented reality. In 2009 International Conference on CyberWorlds, pages 13–20, Sept 2009. – reference: I. Mal, D. Sedlek, and P. Leit£o. Augmented reality experiments with industrial robot in industry 4.0 environment. In 2016 IEEE 14th International Conference on Industrial Informatics (INDIN), pages 176–181, July 2016. – reference: T. Pettersen, J. Pretlove, C. Skourup, T. Engedal, and T. Lokstad. Augmented reality for programming industrial robots. In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings., pages 319–320, Oct 2003. – reference: C. L. Ng, T. C. Ng, T. A. N. Nguyen, G. Yang, and W. Chen. Intuitive robot tool path teaching using laser and camera in augmented reality environment. In 2010 11th International Conference on Control Automation Robotics Vision, pages 114–119, Dec 2010. – reference: S. M. Abbas, S. Hassan, and J. Yun. Augmented reality based teaching pendant for industrial robot. In 2012 12th International Conference on Control, Automation and Systems, pages 2210–2213, Oct 2012. – reference: G. Reinhart, W. Vogl, and I. Kresse. A projection-based user interface for industrial robots. In 2007 IEEE Symposium on Virtual Environments, Human-Computer Interfaces and Measurement Systems, pages 67– 71, June 2007. – reference: R. Bischoff and A. Kazi. Perspectives on augmented reality based human-robot interaction with industrial robots. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), volume 4, pages 3226–3231 vol.4, Sept 2004. – reference: Y. Lin, S. Song, and M. Q. H. Meng. The implementation of augmented reality in a robotic teleoperation system. In 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR), pages 134–139, June 2016. – reference: Jan Guhl, Son Tung Nguyen, and Jörg Krüger. Concept and architecture for programming industrial robots using augmented reality with mobile devices like microsoft hololens. In Emerging Technologies And Factory Automation, USA, 2017. IEEE. – reference: Z. Pan, J. Polden, N. Larkin, S. V. Duin, and J. Norrish. Recent progress on programming methods for industrial robots. In ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics), pages 1–8, June 2010. – reference: P. H. Chiu, P. H. Tseng, and K. T. Feng. Cloud computing based mobile augmented reality interactive system. In 2014 IEEE Wireless Communications and Networking Conference (WCNC), pages 3320–3325, April 2014. – ident: 10.1016/j.procir.2018.01.036_bib0001 doi: 10.1109/ETFA.2017.8247749 – ident: 10.1016/j.procir.2018.01.036_bib00010 doi: 10.1109/WCNC.2014.6953084 – ident: 10.1016/j.procir.2018.01.036_bib00011 doi: 10.1109/INDIN.2016.7819154 – ident: 10.1016/j.procir.2018.01.036_bib0009 – ident: 10.1016/j.procir.2018.01.036_bib0004 doi: 10.1109/IROS.2004.1389914 – ident: 10.1016/j.procir.2018.01.036_bib0005 – ident: 10.1016/j.procir.2018.01.036_bib0003 doi: 10.1109/ISMAR.2003.1240739 – ident: 10.1016/j.procir.2018.01.036_bib0007 doi: 10.1109/CW.2009.14 – ident: 10.1016/j.procir.2018.01.036_bib0008 doi: 10.1109/ICARCV.2010.5707399 – ident: 10.1016/j.procir.2018.01.036_bib0002 doi: 10.1109/ROBOT.2002.1013644 – ident: 10.1016/j.procir.2018.01.036_bib00012 doi: 10.1109/RCAR.2016.7784014 – ident: 10.1016/j.procir.2018.01.036_bib0006 doi: 10.1109/VECIMS.2007.4373930 |
| SSID | ssj0000740312 |
| Score | 2.1477344 |
| Snippet | The increasing complexity of products and machines as well as short production cycles with small lot sizes present great challenges to production industry.... |
| SourceID | crossref elsevier |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 161 |
| SubjectTerms | Augmented Reality Industrial Robotics Robot Programming |
| Title | Intuitive robot programming through environment perception, augmented reality simulation and automated program verification |
| URI | https://dx.doi.org/10.1016/j.procir.2018.01.036 |
| Volume | 76 |
| WOSCitedRecordID | wos000547342800030&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVHPJ databaseName: ROAD: Directory of Open Access Scholarly Resources customDbUrl: eissn: 2212-8271 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0000740312 issn: 2212-8271 databaseCode: M~E dateStart: 20120101 isFulltext: true titleUrlDefault: https://road.issn.org providerName: ISSN International Centre |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Lb9QwELaWwgEOqLxEgSIfuJWgJE78OJYVVQuoQqiU3iLHdqqtutlqm61WQuKf8F8ZP-KN6KrQA5cosZLJbubTzGd7Hgi9kSWROlV10pCySQqgb4kgUiQlJVoIbRri2rcdf2aHh_zkRHwZjX71uTBX56xt-XIpLv6rqmEMlG1TZ2-h7igUBuAclA5HUDsc_0nxB-BFfEDQfFbPuj4Ca-rzonxXnkF6my1cHEJbXCTn4tSV6bRJLZ6iX06mocWXL-y66GbAco3uBe_AF7LxRisVB67rchAAfjvjg68xi-y7tPv_0741M8wDIqk_Dk3dd5cmhn18cjv578enHlkf3RUNRbnCYsXQsubgIxOe-34r78yasWCO2dCeZjQbuObMN2i5ZvX9AsSZ9TlqYou8ZtzVYiVrimz_4fxiSGIf7XZWeSmVlVKlWQVS7qC7OYOZl40M_blawgP2BSbR7lPFP9InZ7oIwus_Zz35GRCao030MMxE8K5H0CM0Mu1j9GBQn_IJ-hGxhB2W8ABLOGAJD7CEV1h6iyOScEASXiEJA5JwRFIvFg-R9BR92_twNN5PQreORAGJ7xJGtOGlKakUhjbgQ2umqQJ2ChN8qgwFZl0qo5gktUhN0RRFwXOhdNqkLK8zSZ6hjXbWmucI66zkRBNap4oWMpdcMKJ0XTPg67zmbAuR_jNWKpSytx1VzqubtLiFkvjUhS_l8pf7Wa-hKtBRTzMrwN2NT7645Zteovv2yq_svUIb3XxhttE9ddVNLuevHeh-AxcRs84 |
| linkProvider | ISSN International Centre |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Intuitive+robot+programming+through+environment+perception%2C+augmented+reality+simulation+and+automated+program+verification&rft.jtitle=Procedia+CIRP&rft.au=Wassermann%2C+Jonas&rft.au=Vick%2C+Axel&rft.au=Kr%C3%BCger%2C+J%C3%B6rg&rft.date=2018&rft.issn=2212-8271&rft.eissn=2212-8271&rft.volume=76&rft.spage=161&rft.epage=166&rft_id=info:doi/10.1016%2Fj.procir.2018.01.036&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_procir_2018_01_036 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2212-8271&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2212-8271&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2212-8271&client=summon |