Repository navigation
LLM-based tagging services - #351
lmrodriguezr wants to merge 3 commits into
Conversation
vangberg
left a comment
vangberg
left a comment
There was a problem hiding this comment.
Nice!
- I think it would be nice to generate the list of allowed tags by storing their description in the database.
- Using structured outputs from Claude, the code can be simplified a lot. See individual comments.
- When we have a good number of human-approved tags, we should add an eval, that scores the suggestions against the human-approved tags. That would make it easy to evaluate different models/see if prompt changes improve accuracy.
| ## Tag definitions | ||
|
|
||
| ### Umbrella tags (assign the umbrella whenever you assign a sub-tag) | ||
| - geographic: name references a place or environment |
There was a problem hiding this comment.
We could add description to Tag and generate this list automatically.
There was a problem hiding this comment.
I missed that tags already have their descriptions in the database, because it was a rich text association. Then I guess we should generate it automatically ;)
| Rank: #{name.rank} | ||
| Etymology: #{etym} | ||
|
|
||
| ## Required response format |
There was a problem hiding this comment.
With structured outputs you can force Claude to conform to a given schema.
| end | ||
|
|
||
| def parse_response(body) | ||
| # The text may not be the first content block (e.g., after thinking) |
There was a problem hiding this comment.
See previous comment about structured outputs. Then you get a guaranteed JSON response with specified format.
| end | ||
|
|
||
| def filter_proposals(proposals) | ||
| proposals.select { |p| valid_slugs.include?(p['slug']) } |
There was a problem hiding this comment.
Setting slug to an enum structured output ensures that Claude only returns allowed tags:
"slug": {
"type": "string",
"enum": ["eponym", "geography", "person", "other", …]
}
|
I think this could be a great fit for Jev. When we have some real data to evaluate against, I will give that a try :) |
That sounds great! Yes! |
A backend-only (rake task) system to propose tags via Anthropic API, as well as a front-end system for curators to review LLM-based tag proposals