AskIt: Unified Programming Interface for Programming with Large Language Models
Large Language Models (LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating adeptness across numerous tasks, from text summarization to code generation. While these abilities open up novel avenues in software design and crafting, their incorporation presents substantial chall...
Gespeichert in:
| Veröffentlicht in: | Proceedings / International Symposium on Code Generation and Optimization S. 41 - 54 |
|---|---|
| Hauptverfasser: | , |
| Format: | Tagungsbericht |
| Sprache: | Englisch |
| Veröffentlicht: |
IEEE
02.03.2024
|
| Schlagworte: | |
| ISSN: | 2643-2838 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Large Language Models (LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating adeptness across numerous tasks, from text summarization to code generation. While these abilities open up novel avenues in software design and crafting, their incorporation presents substantial challenges. Developers face decisions regarding the use of LLMs for directly performing tasks within applications as well as for generating and executing code to accomplish these tasks. Moreover, effective prompt design becomes a critical concern, given the necessity of extracting data from natural language outputs. To address these complexities, this paper introduces AskIt, a domain-specific language (DSL) specifically designed for LLMs. AskIt simplifies LLM integration by providing a unified interface that not only allows for direct task execution using LLMs but also supports the entire cycle of code generation and execution. This dual capability is achieved through (1) type-guided output control, (2) template-based function definitions, and (3) prompt generation for both usage modes. Our evaluations underscore AskIt's effectiveness. Across 50 tasks, AskIt generated concise prompts, achieving a 16.14 % reduction in prompt length compared to benchmarks. Additionally, by enabling a seamless transition between using LLMs directly in applications and for generating code, AskIt achieved significant efficiency improvements, as observed in our GSM8K benchmark experiments. The implementations of AskIt in TypeScript and Python are available at https://github.com/katsumiok/ts-askit and https://github.com/katsumiok/pyaskit, respectively. |
|---|---|
| AbstractList | Large Language Models (LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating adeptness across numerous tasks, from text summarization to code generation. While these abilities open up novel avenues in software design and crafting, their incorporation presents substantial challenges. Developers face decisions regarding the use of LLMs for directly performing tasks within applications as well as for generating and executing code to accomplish these tasks. Moreover, effective prompt design becomes a critical concern, given the necessity of extracting data from natural language outputs. To address these complexities, this paper introduces AskIt, a domain-specific language (DSL) specifically designed for LLMs. AskIt simplifies LLM integration by providing a unified interface that not only allows for direct task execution using LLMs but also supports the entire cycle of code generation and execution. This dual capability is achieved through (1) type-guided output control, (2) template-based function definitions, and (3) prompt generation for both usage modes. Our evaluations underscore AskIt's effectiveness. Across 50 tasks, AskIt generated concise prompts, achieving a 16.14 % reduction in prompt length compared to benchmarks. Additionally, by enabling a seamless transition between using LLMs directly in applications and for generating code, AskIt achieved significant efficiency improvements, as observed in our GSM8K benchmark experiments. The implementations of AskIt in TypeScript and Python are available at https://github.com/katsumiok/ts-askit and https://github.com/katsumiok/pyaskit, respectively. |
| Author | Amarasinghe, Saman Okuda, Katsumi |
| Author_xml | – sequence: 1 givenname: Katsumi surname: Okuda fullname: Okuda, Katsumi email: okuda@csail.mit.edu organization: CSAIL, MIT,Cambridge,USA – sequence: 2 givenname: Saman surname: Amarasinghe fullname: Amarasinghe, Saman email: saman@csail.mit.edu organization: CSAIL, MIT,Cambridge,USA |
| BookMark | eNpVkM9Kw0AYxFdRsK19A5G8QOK3f7PrrQSthUg82HPZ7H6J0WYjm4j49gbUg5eZgR8zh1mSszAEJOSaQkYpmJtiW8lcccgYMJFREEJoDidkbXKjuQRuJBhzShZMCZ4yzfUFWY7jKwDLBeULUm3Gt910m-xD13Tok6c4tNH2fRfaZBcmjI11mDRD_Ec-u-klKW1scdbQftg5PA4ej-MlOW_sccT1r6_I_v7uuXhIy2q7KzZlapmBKW3AO6us9sDq3HGE3Bm0XmmrqNNG6loaLZ2CxubAGBWK1nPHeFReMsn4ilz97HaIeHiPXW_j1-HvAP4NPHJRiA |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IL CBEJK RIE RIL |
| DOI | 10.1109/CGO57630.2024.10444830 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP All) 1998-Present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Computer Science |
| EISBN | 9798350395099 |
| EISSN | 2643-2838 |
| EndPage | 54 |
| ExternalDocumentID | 10444830 |
| Genre | orig-research |
| GroupedDBID | 29O 6IE 6IF 6IK 6IL 6IN AAJGR ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IPLJI OCL RIE RIL |
| ID | FETCH-LOGICAL-a290t-f0dca6a8d02b7c3e07c9ead68a61c8958b5985c60fa70221461bf0d9de6d52523 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 5 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001179185400004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:18:23 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a290t-f0dca6a8d02b7c3e07c9ead68a61c8958b5985c60fa70221461bf0d9de6d52523 |
| PageCount | 14 |
| ParticipantIDs | ieee_primary_10444830 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-March-2 |
| PublicationDateYYYYMMDD | 2024-03-02 |
| PublicationDate_xml | – month: 03 year: 2024 text: 2024-March-2 day: 02 |
| PublicationDecade | 2020 |
| PublicationTitle | Proceedings / International Symposium on Code Generation and Optimization |
| PublicationTitleAbbrev | CGO |
| PublicationYear | 2024 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0027413 ssib057256076 |
| Score | 2.2954721 |
| Snippet | Large Language Models (LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating adeptness across numerous tasks, from text summarization to... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 41 |
| SubjectTerms | artificial intelligence Benchmark testing code generation Codes domain specific language Domain specific languages DSL large language model Programming Python software engineering Task analysis |
| Title | AskIt: Unified Programming Interface for Programming with Large Language Models |
| URI | https://ieeexplore.ieee.org/document/10444830 |
| WOSCitedRecordID | wos001179185400004&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV07T8MwELagYmAqjyLe8sCaYjtxbLOhipeE2g4gdatcPyQEtKhN-f3cuUlRBwaWKEp0VnS2dd93vstHyFWwiNNjkXGrgaAEKzMDRChTCGeVKI01MYlNqH5fj0ZmWDerp16YEEIqPgtdvE1n-X7mlpgqgx1eAJvIgaFvK6VWzVrN4pEKgzfG1jXb4nndEsyZue49DABa5wwooSi6zUgbmioppNy3__kxe6Tz25xHh-uws0-2wvSAtBt1Blpv1kMyuF28P1U3FFBlBJyJNliJ9QlGNOUBo4WRALNuvMG8LH3G8nC4rlKZFPXSPhYd8np_99J7zGr5hMwKw6osMu9sabVnYqJcHphyBtZNqW3JnTZST6TR0pUsWgWRHAW-J2BjfCi9FEBQj0hrOpuGY0K1B2cGxx2PsYgO7DxXOgonbcy1Dyekgw4af63-kDFufHP6x_MzsovTkGq5xDlpVfNluCA77rt6W8wv07z-ALhRodc |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LTwIxEG4MmugJHxjf9uB1se1ut603Q1SICBww4UZKH4lRwcDi73dadjEcPHjZbHYzzWbaZr5vOrMfQjdOB5zus4RqCQTFaZ4oIEKJCHBWsFxp5aPYhOj15GikBmWzeuyFcc7F4jPXDLfxLN_OzDKkymCHZ8AmUmDo2zzLGF21a1XLh4sQvkN0XfMtmpZNwZSo29ZTH8B1SoAUsqxZjbWhqhKDymP9n5-zjxq_7Xl4sA48B2jLTQ9RvdJnwOV2PUL9-8V7p7jDgCs9IM1gE2qxPsEIx0yg1zASoNaNNyEzi7uhQByuq2QmDoppH4sGen18GLbaSSmgkGimSJF4Yo3OtbSETYRJHRFGwcrJpc6pkYrLCVeSm5x4LSCWB4nvCdgo63LLGVDUY1SbzqbuBGFpwZnOUEO9z7wBO0uF9Mxw7VNp3SlqBAeNv1b_yBhXvjn74_k12m0PX7rjbqf3fI72wpTEyi52gWrFfOku0Y75Lt4W86s4xz9TdqUe |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+%2F+International+Symposium+on+Code+Generation+and+Optimization&rft.atitle=AskIt%3A+Unified+Programming+Interface+for+Programming+with+Large+Language+Models&rft.au=Okuda%2C+Katsumi&rft.au=Amarasinghe%2C+Saman&rft.date=2024-03-02&rft.pub=IEEE&rft.eissn=2643-2838&rft.spage=41&rft.epage=54&rft_id=info:doi/10.1109%2FCGO57630.2024.10444830&rft.externalDocID=10444830 |