Repository navigation
Hash cached messages instead of storing raw content for spam detection #121
Description
The in-memory spam-detection cache (MessageCache in src/util/cache/recent-message-store.ts) currently retains up to 5,000 full discord.js Message objects, including their raw text content, purely so isDuplicate/isCrossPost (src/features/spam-detection/detectors.ts) can compare messages for similarity. Storing full message text increases the bot's memory footprint and keeps users' message content resident in the process longer than needed. Replacing the stored content with a hash/fingerprint would shrink memory usage and mean the bot no longer retains readable message text for messages that aren't currently flagged.
Acceptance Criteria
MessageCacheno longer retains the rawcontentstring of cached messages; a hash/fingerprint of the content is stored instead, alongside the existing metadata (id, author, channelId, createdTimestamp) still needed by the frequency and cross-channel rules.isDuplicate/isCrossPostare updated to compare the stored fingerprint instead of the rawcontentstrings.- All existing spam-detection rules (banned tags, invites, spoiler hack, cross-posting, high frequency) continue to work and their existing tests still pass.
- Unit tests are added/updated covering fingerprint-based duplicate detection, including near-duplicate messages (reordered/slightly altered) that should still be flagged as similar.
Open Questions
isDuplicatecurrently uses Jaccard similarity on raw text specifically to catch reordered or slightly altered duplicates, not just exact matches. A plain cryptographic hash (e.g. SHA-256) only supports exact-match comparison and would lose that behavior. Should we use a similarity-preserving fingerprint (e.g. SimHash/MinHash) to keep near-duplicate detection, or is exact-match-only detection acceptable for this change?- Should the fingerprint be computed over the raw message content, or over normalized content (lowercased/trimmed, as
isDuplicatealready does before comparing)?
Out of Scope
- The other content-based rules (banned tags, Discord invites, spoiler hack), which inspect the incoming message directly rather than cached content — they don't store content and are unaffected.
- The moderation log message (src/features/spam-detection/logs.ts), which posts the flagged message's live content to the mod-log channel at the moment a rule fires — that uses the in-flight
Messageobject, not the retained cache, so it's unaffected by this change.
Activity
michal-skraburski commentedon Sep 28, 2026
Looking at your open questions, which seem rather similar, I was already thinking we could normalize the message prior to hashing so that similar looking input matches the same hash.
Normalization Solution
I believe the bot already does a form of this:
"FREE Nitro!! claim here <@everyone> https://ac.b.scam.com/weird"
"free nitro \u200B <@Serverowner> http://k.scam.com/ok"
collide to the hash of "free nitro claim here scam.com".
Open ideas
We could take this idea a step further:
These have the caveats that 1. needs careful consideration around non-latin character sets, and 2. won't work with non-english languages.
Addressing cryptographic hash choice
Given that we're already normalizing prior to hashing a spam message, it may not be necessary to implement a complicated algorithm such as SimHash or a simpler one like MinHash. I'll be frank -- I've never seen either of these algorithms before I researching them now, so I can't in full confidence recommend which one to use, except list the findings I made using published papers.
Firstly, since we are already normalizing the message, a SHA-256, would be probably OK to use instead of a resemblance (jaccard) or cosine similarity algorithm. The messages are pre-stripped of any typically benign content such as extra !!!, tags, domain nesting or paths in urls and more.
MinHash or SimHash?
Please note that the following outlook is based on papers/forums using extremely large sets of content, as SimHash and MinHash is typically used for large document and web page similarity in use cases such as weeding out duplicates for AI training, search engine suggestions, and more.
Choosing the most accurate algorithm (sim/min hash) to our use case is important because the further we utilize an algorithm the worse our verification can become [1].
Out of the two listed algorithms you suggested, MinHash seems a strongly preferred choice over SimHash, in terms of similarity the MinHash algorithm can identify greatly different but similar content while the SimHash allows very few degrees of difference. This means that SimHash could misjudge content that is even 2% different. This makes SimHash almost a SHA-256 algorithm at our scale where messages span only about 25 words at most, meaning SimHash an impractical and poor fit for our use case [2]. This paper [3] granted to me by Claude seems to imply that at 25 words or so with 1 or 2 word changes would give large ratios which would trip the SimHash algo. into misjudging.
Minwise hashing is extremely similar to what we have now, as it measures the same thing we do right now, Jaccaard similarity.
It works like this:
To compare two sets, count the positions where their signatures hold the same value and divide by k. Each position matches with probability equal to the sets' Jaccard similarity, so that fraction is an estimate of it. A larger k gives a more accurate estimate.
Furthermore, I had Claude generate a script which simulates the error rate across increasing k values [4]. Essentially, it seems to suggest that the higher the K value the less wrong judgements are made, however k=16 is far too coarse and low to grant accurate judgements, where at the threshold of 0.78 for k=16 it had a wrong decision rate of 52%, where k=64 had 28%. Anything higher than 64 almost always has diminishing returns, where k=16 is equal to about 64bytes, k=64 is 256b, and k=128 is 512b.
So, assuming that a 125 character V8 UTF-16 (maximum) is roughly 250 bytes, any message that is less than 125 characters is not worth hashing in this system, because it will consume more memory than the current implementation. If the message is in UTF-8, the cost is even worse with minimum 250 characters being the requirement for the hashing to be worth it.
Therefore, while MinHashing would keep the functionality, it is not the solution to the original problem you wanted to solve. I can't recommend that MinHashing would be technically worth it. (With this in mind, reflecting on why there is so little data on short length use with this algorithm makes complete sense.)
Final recommendation
Finally, after concluding that SimHashing and MinHashing is not worth it, we can look at SHA algorithms.
Considering other variants of SHA, we can use SHA-256 but also SHA-512, they use 32 bytes and 64 bytes respectively.
SHA-512 runs faster due to modern CPU architecture, however SHA-256, depending on low level implementation, can save memory by 50% if supported.
Therefore, SHA-256 is the way forward if you are okay with losing out on similarity comparison features such as ignoring order, shingle swapping etc.
[1] www.ediscovery-automation.org/deduplication-family-grouping/near-duplicate-detection/simhash-vs-minhash-for-textual-near-dupes/
[2] https://stackoverflow.com/questions/27712472/choosing-between-simhash-and-minhash-for-a-production-system
[3] Moses S. Charikar, Princeton University 2002, https://dl.acm.org/doi/epdf/10.1145/509907.509965
[4] https://gist.github.com/michal-skraburski/1223983f85aed98a9d88b7c469ee85fe
Further reading
https://arxiv.org/pdf/1706.05698
https://proceedings.mlr.press/v33/shrivastava14.pdf
michal-skraburski commentedon Sep 28, 2026
Also, could we consider possibly hashing reported images such as the Mr Beast screenshot? It might be pretty effective!
wiktoriavh commentedon Sep 28, 2026
Let's go with MinHash, as it keeps the comparison.
hmd-ali commentedon Sep 30, 2026
discord.js already caches messages in
channel.messages.cache, andmessageCreateemits that same instance.MessageCachecurrently stores references to those objects, not copies. Replacing them with a new fingerprint record adds an allocation for every message the client still holds, and readable content stays resident in the process either way. Memory is only saved for messages the client has already evicted. To meet the privacy goal we would likely also need to shrink the client'sMessageManagercache, which affects other features relying onchannel.message.cacheBefore implementing, a heap snapshot comparing the retained size of
MessageCacheagainst theMessageManagercaches would show how many messages we hold alone.