Streaming semantic text deduplicator using Jaccard token overlap and sliding window hashing to eliminate redundant search snippets
-
Updated
Oct 24, 2026 - Python
Streaming semantic text deduplicator using Jaccard token overlap and sliding window hashing to eliminate redundant search snippets
Streaming semantic text deduplicator using Jaccard token overlap and sliding window hashing to eliminate redundant search snippets
A highly optimized, thread-safe, zero-latency streaming text interceptor and sensitive words filter for LLM outputs. Supports Python dicts, raw text, and OpenAI SSE streams out-of-the-box. 专为大模型(LLM)流式输出设计的高性能、线程安全、零延迟文本拦截器与敏感词过滤清洗工具。支持原始文本、Python字典、以及 OpenAI SSE 流。
To associate your repository with the stream-filter topic, visit your repo's landing page and select "manage topics."