mirror of
https://ghfast.top/https://github.com/discourse/discourse-ai.git
synced 2026-07-16 11:36:23 +08:00
e76fc77189
* Minor... use username suggester in case username already exists * FIX: ensure we truncate long prompts Previously we 1. Used raw length instead of token counts for counting length 2. We totally dropped a prompt if it was too long New implementation will truncate "raw" if it gets too long maintaining meaning.
17 lines
351 B
Ruby
17 lines
351 B
Ruby
# frozen_string_literal: true
|
|
|
|
module DiscourseAi
|
|
class Tokenizer
|
|
def self.tokenizer
|
|
@@tokenizer ||=
|
|
Tokenizers.from_file("./plugins/discourse-ai/tokenizers/bert-base-uncased.json")
|
|
end
|
|
|
|
def self.tokenize(text)
|
|
tokenizer.encode(text).tokens
|
|
end
|
|
def self.size(text)
|
|
tokenize(text).size
|
|
end
|
|
end
|
|
end
|
