Email Filters Catch Dolphins Along With Sharks


What's the point in spending hours preparing a newsletter, message or report if it's automatically filtered into the junk folder before the recipient even sees it?

Spam threatens to choke the communication channels promising global freedom of expression.

Internet Service Providers (ISPs), corporate server administrators and end users are increasingly using new anti-spam technology to try to stem the relentless tide of junk email flooding the Net.

The problem is: how can we prevent the dolphins from being caught along with the sharks?

The origin of 'spam'

SPAM is a pink canned luncheon meat immortalised in Monty Python's spam-loving Vikings sketch.

In an Internet context, lowercase spam refers to unsolicited commercial or bulk email (such as get-rich-quick schemes, miracle cures, weight loss, Viagra, lotteries, loans, pornography and Nigerian sob stories) and allegedly originated in a MUD/MUSH community.

Of more practical use is the origin of the actual spam mail itself.

Where does all the junk come from?

In the mid-90s, Usenet newsgroups (also called "discussion groups" or "bulletin boards") were the number one source of email addresses for spammers.

Today, the most common origin is web pages, especially if they're listed in a search engine or directory.

Some people have tried foiling address-seeking spambots by inserting the word "UNSPAM" in capitals in the middle of all email addresses on their sites. This stops auto spammers working but enables human beings to work out what to do.

Spammers also harvest addresses from headers of messages you send to friends who forward them to their friends (a good reason for using BCC -- blind carbon copy rather than simple CC which displays all recipients - although some people filter out mail sent using BCC as many spammers also use it).

Other sources include open e-mail discussion lists and web pages that invite you to "insert your address here to be on a 'do not mail' list".

Spammers can simply guess addresses by generating lists of popular names and random words attached to common domains (bob@aol.com, john@hotmail.com).

Once on a spam list, the only way to get off is to change addresses.

If you reply or respond to instructions to "remove", your message will simply confirm your address is valid and you'll get even more junk.

Depending on your email client, you can try tracing junk back to its owner by contacting the server listed in the full message header information (the From address is generally fake - check your Help files to find out how to "reveal full headers").

How to stop spam

Despite legislation against unsolicited commercial email, the volume of junk is increasing alarmingly.

The simplistic oft-cited fix -- "just hit delete" -- is only a bandaid solution and fails to discourage the junk merchants.

Self-regulation and industry codes are difficult to enforce. ISPs face problems if they disconnect service to spammers under some countries' telecommunications laws.

Technical solutions have centred on filtering technology.

Types of filters

Many corporations and ISPs filter incoming mail on or after delivery.

Server-side filtering software typically looks at the headers, subject line and/or contents of the message.

Some filters -- and their users -- are smarter than others.

SpamAssassin is an open-source, collaborative, community anti-spam effort based on filtering rules to analyse email content.

The software gives each message a score based on how many rules it breaks.

Any programmer can suggest rules for new releases of the software which spots, not blocks, spam.

ISPs and server administrators then decide whether to send suspect mail to junk folders, automatically delete mail tagged as spam, or bounce it back to sender.

Unfortunately for email publishers, some of the filter rules are too broad or the threshold is set too low.

Many innocent messages are being lumped in with the guilty.

One of my newsletter readers notified me that his ISP had tagged a recent issue as spam -

SPAM: -------------------- Start SpamAssassin results ----------------------
SPAM: This mail is probably spam. The original message has been altered
SPAM: so you can recognise or block similar unwanted mail in future.
SPAM: See http://spamassassin.org/tag/ for more details.
SPAM:
SPAM: Content analysis details:   (13.2 hits, 6 required)
SPAM: Hit! (1.5 points)  BODY: Asks you to click below
SPAM: Hit! (0.2 points)  BODY: No such thing as a free lunch (1)
SPAM: Hit! (2.6 points)  BODY: Instant Access button
SPAM: Hit! (3.5 points)  URI: URL of page called "unsubscribe"
SPAM: Hit! (4.1 points)  URI: 'remove' URL contains an email address
SPAM: Hit! (2.1 points)  BODY: FONT Size +2 and up or 3 and up
SPAM: Hit! (0.0 points)  BODY: Includes a URL link to send an email
SPAM: Hit! (-0.8 points) BODY: Image tag with an ID code to identify you
SPAM:
SPAM: -------------------- End of SpamAssassin results ---------------------
 
Even if you can identify the problems that raised your score, you can do little about it.

Best practice in e-newsletter publishing -- and now law in many countries -- is to enable your readers to unsubscribe easily yet the word "unsubscribe" or similar would lead to your newsletter being blocked by filters such as SpamAssassin.

Just look at the seemingly innocuous words on the constantly changing list of tests SpamAssassin performs on mail messages to determine whether they're spam.
 
A growing number of email users also filter their own inboxes by setting up rules to delete messages containing certain keywords or punctuation (for instance, the phrase, "This is not spam", which is among the most commonly used phrase in junk email).

Some desktop applications, such as MailShield and MailWasher, pre-set the rules and tag or delete mail without the user having to do anything.
 
Some client-side filters prioritise mail into different folders.

To ensure they receive wanted mail, email users can add favourite mailing lists to a "white" or "green" list so the mail ends up in a specific folder.

The problem with email filtering

Arbitrary and unpredictable email filtering, as with web content filtering, can backfire and cost time, money and frustration -- the very things filters were designed to save.

Tracking down problems and dealing with aggravated users or administrators wastes resources that could be spent creating content, marketing and improving customer service.

Many filters can be imposed only after you have received unsolicited email from each source, meaning you have to respond manually to every single spammer.

Spammers regularly change their email addresses to bypass filters.

"Qualitative" filters can try to detect junk mail from unknown sources but legitimate mail can be inadvertently trashed in the process ("false positives") while some junk mail still gets through ("false negatives").

Filters are becoming a serious problem with many legitimate opt-in publishers being mistaken for spammers and blacklisted.

Once on a black list, the only way to be unblocked is to contact the ISP and ask to be placed on a white list.

While some e-newsletter publishing tools let you monitor how many messages are opened or bounced, you might never know which readers are being filtered unless you know which email program they're using.

Even then, the recipient might just be busy, uninterested, on leave or not receiving your message.

Some filter programs don't even notify you that your message was blocked while others bounce back mail with "sensitive" content without specifying the rejected words.

On another occasion my own free weekly e-newsletter was blocked on its way to at least two subscribers thanks to Trend SMEX Content Filter detecting "sensitive content" yet the "naughtiest" word I could find was "teenage".

Pity the poor medical publishers who often use  "controversial" words such as "breast" or "penis" in their newsletters.

You might be able to track down the reason you were blocked by examining the full header information (ask your subscribers to forward the entire copy of your junked message).

The header should list filter rules that the message violated, if the ISP uses software such as SpamAssassin.

If not filtering, then what?

Some online publishers advocate web services where you submit your newsletters for screening of content or technical problems that might trigger spam filters before mailing.

Of course, spammers could also abuse such a service to get their own mail past the filters.

Others back the idea of an Internet-wide white list along the lines of web site privacy site TRUSTe.

Spam-busting campaigners believe the only effective method of addressing the problem is through simple, direct legislation to:

  • Ban the practice of transmitting unsolicited bulk email
  • Ban the sale and marketing of spamming tools
  • Provide a right of action and damages to the recipient
  • Express damages in terms of penalty units
  • Provide for damages to be trebled for wanton violations
  • Cover offshore spamming by local interests, and
  • Attach to the outgoing revenue stream of foreign interests, which spam locals for the benefit of the spammers' revenue stream.

Meanwhile, ashile the world waits for more countries to introduce tough anti-spam legislation, online publishers must continue to be vigilant.

How to avoid the junk folder

Choose the subject line and words in your emails carefully if you don't want to end up in the junk file.

More people are setting up their email programs to filter incoming mail automatically for classic junk mail keywords and symbols (eg "advertisement", "free", "rich", "porn", "lover,", "opt-in,"  "e-mail", "money-back guarantee", "teen", "sex", ! # or $, removal instructions, all capitals).

Drop all hype and advertising-speak from your subject lines and messages.

Apart from saving you from the filters, honest concise text is preferred by the impatient, overloaded online reader.

Online publishers wishing to avoid being trashed should send a welcome message to new subscribers or correspondents telling them how your messages will appear (eg "the newsletter will appear from yvette@brizcomm.com.au") and suggesting they add the address to the white or green list of their email program.

Set up test accounts with popular email providers such as AOL, Yahoo and HotMail to see if your messages are getting through.

Beware Microsoft Outlook's "junk e-mail filters" which, when turned on, colour or delete messages considered spam based on a filters.txt file.

I searched for the filters.txt file on my PC to see which words were no-gos and was surprised to see so many seemingly harmless words.

Use a professional list hosting service unless you can afford an experienced in-house support staff.

E-newsletter publishing is becoming a highly specialised and complex field - don't risk your reputation to amateurs.

Ensure your list host monitors other clients to ensure they're publishing to only opt-in recipients -- if opt-out mailers, rental lists and spammers are blacklisted, you might be guilty by association.

Insist on detailed bounce reports for every issue you send out.

You should be able to see which names bounced and the exact reasons (eg "mailbox full", "address does not exist" or the frustrating "unknown").

Just because your bounce report says your mail was delivered, doesn't mean it did.

Many filters stop email after the "not-bounced" signal has been sent.

Monitor your open and clickthrough reports for unusual patterns.

Read all reader feedback, particularly if readers are having trouble subscribing or getting your mail.

Chances are they might have to ask (or scream at) their ISP or techies to add you to a white list.

(Of course, you need to publish quality content if you want your readers to jump up and down when they don't get it.)

Many permission-based publishers include their newsletter or trademark in square brackets in the subject line to avoid being filtered.

Other ethical publishers, including me, have resorted to disguising known filter trigger words by inserting unusual characters or deliberate typos -- for instance, using f^ee instead of free or sp*m instead of spam.

It's just a matter of time before the spammers work around these "tricks".

Bastardising or censoring our language to suit new communications technology would seem an extreme solution -- not to mention bitterly ironic -- but what's the alternative?

Useful links

Coalition Against Unsolicited Commercial Email
http://www.cauce.org/

The Net Abuse FAQ
http://www.cybernothing.org/faqs/net-abuse-faq.html#2.1

How to show full header details for your email program -
http://www.uia.net/emailabuse.htm

Put junk mail in its place (tips on using Outlook) -
http://www.microsoft.com/Office/using/column01.asp

Tips on writing hype-free online content
http://www.brizcomm.com.au/

Yvette Nielsen speaks from hands-on experience as an online content creator, publisher, coach, consultant, trainer, and award-winning writer and sub-editor.

Yvette pioneered web writing training in Australia in 1998 and has inspired thousands of people through her workshops and presentations from London to LA, Sydney to Seoul.

She has reviewed web sites professionally for print, online and radio for since 1997. Her own e-newsletter and site - www.brizcomm.com.au - won the Smart Communities category in the 2002 Asia Pacific ICT Awards.





Latest News:



Site: Yahoo! News Search Results for news
JCDecaux in talks for News Outdoor Group in Russia (AP via Yahoo! News)
French outdoor advertising firm JCDecaux SA said Thursday it is negotiating to buy Russian rival News Outdoor Group from global media company News Corp. News Corp tightens grip on Premiere with new CEO (Reuters via Yahoo! News)
Rupert Murdoch tightened his grip on German pay-TV broadcaster Premiere by appointing News Corp executive Mark Williams as CEO in a move to get the company back on track. JCDecaux and News Corp. in talks to combine outdoor ad groups (International Herald Tribune)
The News Corp. chief executive, Rupert Murdoch, has recently expressed nervousness about investments in Russia, where News Outdoor generated the bulk of its revenue last year. News Corp says Mark Williams interim Premiere CEO (Reuters via Yahoo! News)
News Corp. executive Mark Williams has taken over as chief executive of German pay-TV broadcaster Premiere in an interim capacity, News Corp. said after the move led to renewed takeover speculation. News Corp. says Mark Williams interim Premiere CEO (Reuters via Yahoo! News)
Mark Williams, a News Corp. executive named late on Wednesday as chief executive of German pay-TV broadcaster Premiere, has taken over in an interim capacity, a News Corp. spokeswoman said on Thursday. JCDecaux in talks to buy News Corp. billboard unit (AP via Yahoo! Finance)
In a deal that would create the world's largest outdoor advertising company, French conglomerate JCDecaux SA said Thursday that it is in talks to buy Russian rival News Outdoor Group from global media company News Corp. Weekly world news quiz (BBC News)
At the end of another week, test your knowledge with our world news quiz. UPI NewsTrack Entertainment News (UPI)
Kanye arrested in paparazzi scuffle ... Report: Critic smacked Ebert with binder ... Cafe Tacvba nominated for 6 Latin Grammys ... Groban to perform TV songs at Emmys ... News from United Press International. United shares climb after news debacle (UPI)
CHICAGO, Sept. 11 (UPI) -- A computer program assigning dates to news stories was partly to blame for a sell-off of United Airlines stock Monday, a Google spokesman said. JCDecaux in talks for News Outdoor Group in Russia (San Francisco Chronicle)
French outdoor advertising firm JCDecaux SA said Thursday it is negotiating to buy Russian rival News Outdoor Group from global media company News Corp. In a statement, the global seller of outdoor advertising said it was in exclusive talks with Rupert...

Site: news - Google News
New Voices in Congress Will Change the Tone of the Democratic Majority - New York Times

guardian.co.uk

New Voices in Congress Will Change the Tone of the Democratic Majority
New York Times, United States - 12 hours ago
The latest on the presidential transition and other political news from Washington and around the nation. Join the discussion. ...
Video: Burris Reflects on His Trip to Washington AssociatedPress
Power, pop, and probings from ABC News Senior White House ... ABC News
New Congress hears call to 'get something done' USA Today
The Associated Press - Los Angeles Times
all 9,822 news articles
Ballmer touts Windows 7 beta, new deals - CNET News

PC World

Ballmer touts Windows 7 beta, new deals
CNET News, CA - 1 hour ago
Microsoft will also announce new deals for Windows Live that will see Microsoft's search engine become the default on PCs from Dell as well as touting a ...
Microsoft's Ballmer touts new Windows, search deal MarketWatch
A Smaller CES, But Big Tech News Still Expected CNBC
Five Reasons Microsoft Plans Free Windows 7 Upgrades CRN
all 456 news articles
Phil Schiller bringing the news at Macworld - San Francisco Chronicle

CNN

Phil Schiller bringing the news at Macworld
San Francisco Chronicle,  USA - 4 hours ago
We get going at 9 am and we'll be delivering the news as it happens. 8:53 am Just a few minutes away. Listened to some Killers music on the way in. ...
Video: Steve Jobs Weight Loss Due to Hormone Imbalance AssociatedPress
O'Brien: Apple's final appearance at Macworld signals new era San Jose Mercury News
News from the Mac world New Zealand Herald
Contra Costa Times - Ars Technica
all 3,104 news articles
US Senate Democratic Leaders Outline New Agenda - Voice of America

ABC News

US Senate Democratic Leaders Outline New Agenda
Voice of America - 3 hours ago
By Deborah Tate A day after a new Democratic-led US Congress was sworn in, lawmakers are focusing on an ambitious agenda. ...
Video: Obama to Bring Accountability to Spending Plan AssociatedPress
New Congress arrives with eye on Obama's stimulus package MarketWatch
House Democrats Pass New Rules to Curb Republican Influence CNSNews.com
Milwaukee Journal Sentinel - New York Times
all 6,139 news articles
Sony Ericsson's new W508 Walkman phone - CNET News

CNET News

Sony Ericsson's new W508 Walkman phone
CNET News, CA - 3 hours ago
Sony Ericsson expanded its Walkman phone line at CES with the announcement of its new W508 music phone. Sporting a flip phone design that reminds us of the ...
New Sony Hard-Drive HD Camcorders - PC World

PC World

New Sony Hard-Drive HD Camcorders
PC World - 3 hours ago
The SR line will live on, however, with three new standard-definition units. Pictured above is the Sony Handycam HDR-XR520V ($1500, available in March). ...
Sony Announces Four New High-Definition and Three Standard ... CameraTown.Com (press release)
New Sony Camcorders Do GPS, Too PC World
Sony Announces Two New DVD Camcorders: DCR-DV850 and DCR-DVD650 CameraTown.Com (press release)
PC World - PC World
all 37 news articles
CNET News Daily Podcast: When businesses strike back at Yelpers - CNET News

CNET News Daily Podcast: When businesses strike back at Yelpers
CNET News, CA - 4 hours ago
Also in this podcast: Yahoo takeover talk returns in the new year; Intel revenue drops by $2 billion; and Facebook reaches a new milestone. ...
Pequot Said to Face New SEC Insider-Trading Inquiry - Bloomberg

Pequot Said to Face New SEC Insider-Trading Inquiry
Bloomberg - 6 hours ago
7 (Bloomberg) -- The US Securities and Exchange Commission opened a new investigation into whether Pequot Capital Management Inc., the hedge fund run by ...
Hedge fund Pequot faces new SEC probe - Bloomberg Reuters
all 8 news articles
Israel Mulls New Gaza Cease-Fire Proposal - CBS News

ABC News

Israel Mulls New Gaza Cease-Fire Proposal
CBS News, NY - 17 hours ago
Israel entered the current conflict saying it wanted to change the situation in Gaza, not necessarily change the regime, reported CBS News correspondent ...
Video: Gaza Gets a Brief Break AssociatedPress
News Analysis: How far to Gaza ceasefire? Xinhua
AP Top News at 10:08 am EST The Associated Press
Global Voices Online - The Associated Press
all 29,536 news articles
A roundup of news, schedules, and key stories from CBS News ... - CBS News

BBC News

A roundup of news, schedules, and key stories from CBS News ...
CBS News, NY - 13 hours ago
President-elect Obama holds a news conference today in Washington, DC at 10:15am ET where, per the Wall Street Journal?s Jonathan Weisman, ...
Video: Analyst Weighs in on Obama's CIA Pick AssociatedPress
New administration defends Panetta pick SmartBrief
News roundup: Israel presses on in Gaza; Obama tries to reassure GOP USA Today
Chicago Sun-Times - Huffington Post
all 2,435 news articles

MORE RESOURCES:
Site: Yahoo! News Search Results for ezine publishing
Site: ezine publishing - Google News
The Ezine Queen's 7 Reasons to Grow Your Business with Ezines ... - Gather.com

The Ezine Queen's 7 Reasons to Grow Your Business with Ezines ...
Gather.com, MA - Dec 27, 2008
As one of the leading authorities on email publishing, she is a proud advocate of using the ezine to drive sales. According to Alexandria, offering a free ...
ATTENTION TO THOSE CLAIMING SPAM!!!!!!!! - CrunchGear

ATTENTION TO THOSE CLAIMING SPAM!!!!!!!!
CrunchGear, NY - Jan 6, 2009
If I can post to a site like Ezine Articles and gain and expert status with them, and they generally don?t have a problem with my short articles, ...
Top 3 Ways To Boost Your Affiliate Commissions Overnight - Gather.com

Top 3 Ways To Boost Your Affiliate Commissions Overnight
Gather.com, MA - Jan 3, 2009
Publish an online newsletter or Ezine. It is always best to recommend a product to someone you know than to sell to a stranger. This is the purpose behind ...
New Book Features Emerging Trends for Online Businesses ... - PR Web (press release)

PR Web (press release)

New Book Features Emerging Trends for Online Businesses ...
PR Web (press release), WA - Dec 30, 2008
She publishes a monthly ezine & offers a free Copywriting Action Plan. Visit http://www.LisaManyon.com for more information.
Tracking Your Article Submissions a Way to Maximize Your Traffic ... - PWO

Tracking Your Article Submissions a Way to Maximize Your Traffic ...
PWO, France - Dec 12, 2008
... all the republishing and traffic you will get from article publishing. Among a couple of the top directories watercolor is eZine Articles, Article City, ...
Article Marketing Tips ? How to Outsource for the Big Bucks PWO
all 2 news articles
Home | Site Map | Resource Links