Avatar

Short insight into the spam filtering with the new B8-filter (General)

by Auge ⌂, Friday, May 13, 2022, 11:42 (685 days ago)

Hello

Since 2022-05-08 this forum run with a version with an additional local spam recognition system, named B8. It's a Bayes filter that should classify content as spam or ham (no spam). The filter is based on the work of the 18th century mathematician Thomas Bayes.

Here, in this forum, we use not only this mehtod but also Bad Behavior and the black lists that work both locally as well as the external services, the forum script integrates optionally.

From sunday evening (2022-05-08, circa 20:00 UTC) til now (with the transient outage of the forum) eight entries got classified as spam. None of them was a false positive (a ham entry, that was erroneously classfied as spam). All affected entries was hidden from the forum visitors and the only task for a team member (in this case it was me) was, to manually check every single entry and to confirm (or deny) that it was really spam.

That way the filter learns, that "he" was right what will improve the detection in prospective. It's a bit of regulary work to start with but over the time the filter will get better and better and need less regulary attention.

Nice one!

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

Avatar

Short insight into the spam filtering with the new B8-filter

by Micha ⌂, Friday, May 13, 2022, 11:47 (685 days ago) @ Auge

Hello Auge,

That way the filter learns, that "he" was right what will improve the detection in prospective. It's a bit of regulary work to start with but over the time the filter will get better and better and need less regulary attention.

Yes, I agree. It is important to flag both SPAM entries but also HAM entries. The filter does not train automatically in the forum.

/Micha

--
applied-geodesy.org - OpenSource Least-Squares Adjustment Software for Geodetic Sciences

Avatar

Short insight into the spam filtering with the new B8-filter

by Auge ⌂, Friday, May 13, 2022, 13:42 (685 days ago) @ Micha

Hello Micha

That way the filter learns, that "he" was right what will improve the detection in prospective. It's a bit of regulary work to start with but over the time the filter will get better and better and need less regulary attention.


Yes, I agree. It is important to flag both SPAM entries but also HAM entries. The filter does not train automatically in the forum.

With flagging ham you mean all entries, that are actually not flagged as spam? I ask, because I am a bit confused by the two options to "report and flag as ham" or to only "flag as ham"? What's the difference?

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

Avatar

Short insight into the spam filtering with the new B8-filter

by Micha ⌂, Saturday, May 14, 2022, 07:40 (685 days ago) @ Auge

Hello,

With flagging ham you mean all entries, that are actually not flagged as spam?

Yes.

I ask, because I am a bit confused by the two options to "report and flag as ham" or to only "flag as ham"? What's the difference?

Take a look to the basic equation. The filter needs both, HAM training data and SPAM training data. If you only flag SPAM postings, it is nothing else then a black list. However, the filter evaluates the words in an entry and estimates the probability that these words are often used in SPAM (or HAM) postings.
So, in my opinion, it is not a good choice to use Akismet in parallel with B8, if you like to train the filter.

If you click to "report and flag as ham", the word list of the postings is stored to the database, and the related HAM counter of each word is increased, cf.

mlf2_b8_wordlist --> `token`, `count_ham`, `count_spam`

Using this table one can evaluate whether a single word (the token) occurs more often in SPAM or HAM entries. Since a single word does not allow for a statistically firm decision, the probability of the whole word list is evaluated.

The option "flag as ham" means, that the posting is flagged as HAM but the word list is not used to train/improve the filter (not stored to mlf2_b8_wordlist).

/Micha

--
applied-geodesy.org - OpenSource Least-Squares Adjustment Software for Geodetic Sciences

Avatar

Short insight into the spam filtering with the new B8-filter

by Auge ⌂, Sunday, May 15, 2022, 11:15 (683 days ago) @ Micha

Hello

… flagging ham …

I ask, because I am a bit confused by the two options to "report and flag as ham" or to only "flag as ham"? What's the difference?

If you click to "report and flag as ham", the word list of the postings is stored to the database, and the related HAM counter of each word is increased, cf.

mlf2_b8_wordlist --> `token`, `count_ham`, `count_spam`

Using this table one can evaluate whether a single word (the token) occurs more often in SPAM or HAM entries. Since a single word does not allow for a statistically firm decision, the probability of the whole word list is evaluated.

The option "flag as ham" means, that the posting is flagged as HAM but the word list is not used to train/improve the filter (not stored to mlf2_b8_wordlist).

Thank you for this insight.

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

Avatar

Recommendations for working with the new B8-filter

by Auge ⌂, Thursday, June 23, 2022, 13:01 (644 days ago) @ Auge

Hello

Meanwhile the project forum works since more than six weeks with the new filter system. We encountered a few minor issues and made mainly good experiences with the new system.

1. Issue: unlocking entries

When classifying entries as ham, the entries will be unlocked in the case they was locked before. This occurs in the case of classifying old threads that was locked by hand or by automatisms (after a certain time, after a time of inactivity or after a number of opened newer threads). Currently one can circumvent the problem by locking the thread again after classifying the last entry of the thread as ham. The next minor release will fix this issue.

2. Observation: the bayes filter found many spam entries after a short time of training

From the very first days of bringing the Bayes filter into service we saw most of the spam attempts to be hidden. We see up to 40 proper classified spam entries on some days. In the first days the Bayes filter ran parallel with the Akismet filter. After a few days we disabled the Akismet filter for forum entries. Additionally there is the classical bad word list that is part of the forum software since the very first days (implemented since My Little Forum 1.x).

So the filters in their combination found many typical spam attempts from the beginning. These entries have to be manually classifyed afterwards to train the Bayes filter. This is an ongoing task for the forum team.

After disabling the Akismet filter for forum entries the combination of the Bayes filter and the bad word list finds nearly all of the spam entries but not all. Currently we see none or one to three entries a day that is not properly classified automatically. Classifyiing those spam entries manually is part of the ongoing task for the forum team, mentioned above.

3. Training: diligent work of classifying existing entries as ham

The Bayes filter is in generally not a spam filter but a system that computes probabilities. In our case probabilities for entries being ham or spam. There are words in spam entries, that will occuring with a very low probability in ham entries. On the other hand there are many words, sequences or sentences that one will find here and there. So the filter must get trained for spam but also for recognising ham.

Especially if one is running a well visited forum since a long time, one should have a more or less big amount of entries, that is classifyable as ham – assuming the spam entries will be deleted regulary. Having classified all content of a forum, the Bayes filter will have a proper base to make it's decisions. On the other hand, it will be a very diligent work to classify all the content manually.

We have over 8200 entries in this forum (the project forum). Since the launch of the Bayes filter system I am working on the classification of ancient entries and threads. I often thought about how to automate this process during this work, but until now I had no proper idea how to do it. You have to keep in mind, that you as the forum operator want to check, if an entry is really ham, if it has classifyable content, only a few agreeing or disagreeing words or no content at all but should be kept. For every single case you have to decide if you want to classify the entry as ham and if so, to additionally train the filter or not.

That means, that the work, to look into every single entry, remains. There is no escape but not to use the Bayes filter. But the filter brings you as a forum operator into the position to (at least partly) abandon dependencies on external services. So we strongly recommend to work with the filter and to get down to the business to classify the old entries. Once done, one will have only to do the daily work on current forum traffic. And this is a manageable job (exception proves the rule).

As last: currently we have no experience with the "automated filter training".

Tschö, Auge

--
Trenne niemals Müll, denn er hat nur eine Silbe!

RSS Feed of thread