Other packages > Find by keyword >

rtiktoken  

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View on CRAN: Click here


Download and install rtiktoken package within the R console
Install from CRAN:
install.packages("rtiktoken")

Install from Github:
library("remotes")
install_github("cran/rtiktoken")

Install by package version:
library("remotes")
install_version("rtiktoken", "0.0.6")



Attach the package and use:
library("rtiktoken")
Maintained by
David Zimmermann-Kollenda
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2024-11-06
Latest Update: 2024-11-06
Description:
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
How to cite:
David Zimmermann-Kollenda (2024). rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models. R package version 0.0.6, https://cran.r-project.org/web/packages/rtiktoken. Accessed 23 Nov. 2024.
Previous versions and publish date:
No previous versions
Other packages that cited rtiktoken R package
View rtiktoken citation profile
Other R packages that rtiktoken depends, imports, suggests or enhances
Complete documentation for rtiktoken
Functions, R codes and Examples using the rtiktoken R package
Full rtiktoken package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

Simile  
Interact with Simile Models
Allows a Simile model saved as a compiled binary to be loaded, parameterized, executed and interroga ...
Download / Learn more Package Citations See dependency  
nextGenShinyApps  
Craft Exceptional 'R Shiny' Applications and Dashboards with Novel Responsive Tools
Nove responsive tools for designing and developing 'Shiny' dashboards and applications. The scripts ...
Download / Learn more Package Citations See dependency  
highlight  
Syntax Highlighter
Syntax highlighter for R code based on the results of the R parser. Rendering in HTML and latex mar ...
Download / Learn more Package Citations See dependency  
abcADM  
Fit Accumulated Damage Models and Estimate Reliability using ABC
Estimate parameters of accumulated damage load duration models based on failure time data under a Ba ...
Download / Learn more Package Citations See dependency  
triplot  
Explaining Correlated Features in Machine Learning Models
Tools for exploring effects of correlated features in predictive models. The predict_triplot() func ...
Download / Learn more Package Citations See dependency  
RH2  
DBI/RJDBC Interface to H2 Database
DBI/RJDBC interface to h2 database. h2 version 1.3.175 is included. ...
Download / Learn more Package Citations See dependency  

23,229

R Packages

199,929

Dependencies

62,984

Author Associations

23,230

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA