discourse-mirrors/discourse-ai
0
0
Fork 0
mirror of https://ghfast.top/https://github.com/discourse/discourse-ai.git synced 2026-07-16 11:36:23 +08:00
Code Issues Projects Releases Packages Wiki Activity Actions
discourse-ai/lib/shared/tokenizer
History
Exact
Sam e76fc77189
fixes (#53) 
* Minor... use username suggester in case username already exists

* FIX: ensure we truncate long prompts

Previously we

1. Used raw length instead of token counts for counting length
2. We totally dropped a prompt if it was too long

New implementation will truncate "raw" if it gets too long maintaining
meaning.
2023-05-06 07:31:53 -03:00
..
tokenizer.rb fixes (#53) 2023-05-06 07:31:53 -03:00