Testing Newsletter Subject Lines: What We Learned From 6 Months of A/B Tests

How much of a difference can changing a single word in your email subject line REALLY make?

You might be surprised!

(It still surprises us, sometimes.)

Every week, we send our newsletter to 125,000 + subscribers – and every week, we send it with two different newsletter subject lines. (Before we keep going, here’s the link where you can get on that list.)

One reason we do this is to maximize our open rate for each individual newsletter – a process you can read more about in this post – but the OTHER reason is so we can learn more about email subject line testing and writing better subject lines.

Changing just one or two words in your email subject line can make a HUGE impact on your open rates – and reveal a lot about your readers!

Want to see how?

A while ago, we took a hard look at six months of testing subject lines for our newsletter and found some pretty distinct patterns in which kinds of subject lines score more opens – and which ones are easy for our readers to ignore.

What is an email subject line?

Your email subject line is your chance to make a first impression. It’s the line of text your email subscribers see in their inbox before opening your email. The subject line could be the deciding factor as to whether someone opens your email or not. You should always check the subject line before clicking send on your email!

What have we learned from 6 months of testing newsletter subject lines?

These trends aren’t necessarily true for everyone, but they DO demonstrate the importance of testing your own subject lines – you could learn something that dramatically improves your open rates over time!

The more specific, the better

A long, long time ago, when the Earth was young and Edgar hadn’t yet been created, our founder and CEO Laura Roeder sent out a weekly newsletter to her own audience of social media marketers.

Here are a few actual newsletter subject lines from those emails:

  • my “no-no” face
  • two magic words
  • here’s what to do next
  • the #1 most important factor
  • nicely done!

We will mail you ten thousand dollars if you can guess what ANY of those emails was about.

(Note: we will not mail you ten thousand dollars.)

Laura’s newsletter consistently performed pretty well, which might make you think that the less information you give in the subject line, the better – right? A little curiosity can go a long way!

But not necessarily.

We’ve put that logic to the test with our own newsletter email subject lines, and when it comes to our audience, we’ve found that specificity is more useful than ambiguity.

There’s a good reason for that, and we’re going to talk about it in a second – but first, here are some examples of what we’re talking about.

Subject line comparison for March 14 Newsletter

These email newsletter subject lines aren’t that different – but one big distinction is their degree of specificity.

In Combo 1, “Twitter’s new rules” could mean a lot of different things.

(Maybe they have new guidelines about the appropriate use of reaction GIFs taken from TV’s The Bachelor, which would be highly relevant to our own interests.)

Combo 2, however, is more specific – Twitter’s new rules have to do with social media scheduling. A reader can see that email’s subject line and understand immediately just how relevant it is to them, and that makes the importance of opening it feel much more urgent.

(Here’s where you can read that post, by the way.)

Let’s check out another:

  1. [The MeetEdgar Newsletter] Facebook’s THREE new algorithm updates 👩‍💻
  2. [The MeetEdgar Newsletter] Facebook’s latest algorithm changes explained 👩‍💻

Again, these subject lines are barely different – but Combination 1 took the lead by specifying that Facebook has implemented three algorithm updates, speaking to how comprehensive this post is going to be.

Here are a few more quick examples – see if you can guess the winner for each:

  1. [The MeetEdgar Newsletter] the social media news you might’ve missed 👀
  2. [The MeetEdgar Newsletter] everything you might’ve missed 👀

If you guessed Combination 1, you guessed right! “Everything” isn’t compellingly specific – it could mean pretty much anything. The other version is much clearer about what it offers.

Let’s try one more:

  1. [The MeetEdgar Newsletter] this new Facebook Live feature 👀
  2. [The MeetEdgar Newsletter] Facebook Live’s new screensharing feature 👀

In this case, Combination 2 was the clear winner – and again, it was by offering a specific detail. An unspecified new feature may or may not make someone curious enough to open this email, but specifying the type of update makes a big difference!

Why do we find that specificity works better for us than ambiguity?

Especially when that newsletter from years ago, for example, found so much success being vague?

One answer may simply be that these are two different audiences, who respond differently to a single format.

An equally likely culprit, though, is the fact that vague headlines in general just don’t perform the way they used to.

Being vague and tease-y has fallen out of favor with both audiences and social networks like Facebook. These days, clickbait-style ambiguity is more likely to get you ignored or hidden from news feeds, while giving readers a clear idea of what to expect is more popular than ever.

(Which speaks to the importance of monitoring your success over time – what works for you right now might not be as successful in a year or two!)

Okay, enough specifics about specificity.

What ELSE have we learned from our email subject tests?

Use language with muscle

“Use strong language” is one of those lessons that sounds kind of insultingly obvious – like “don’t season your porterhouse steak with glitter.”

In practice, though, it takes different forms that aren’t always so clear!

And while specificity like we just talked about can make your language stronger, sometimes colloquialisms can give a little extra oomph, too!

Here’s an example:

  1. [The MeetEdgar Newsletter] our most popular sales page tips 🤝
  2. [The MeetEdgar Newsletter] our hottest sales page tips 🔥

Technically, Combination 1 is more specific, but Combination 2 is more exciting. (Especially because “hottest” can indicate both that something is popular and that something is useful.)

Here’s another example of that exact principle in practice:

  1. [The MeetEdgar Newsletter] 10 can’t-miss Facebook strategy guides 💘
  2. [The MeetEdgar Newsletter] roundup: our fave Facebook strategy guides 💘

Combination 1 was the winner here – not only does it give a specific number of guides the reader can expect, but it describes them as “can’t-miss” instead of “our fave.” The former ascribes an intrinsic value to them, which is much stronger than suggesting their worth is dependent on our opinions.

(We won’t take that too personally.)

Here’s one more example of how valuable strong language can be:

  1. [The MeetEdgar Newsletter] Facebook’s “engagement bait” page demotions
  2. [The MeetEdgar Newsletter] Facebook is demoting pages and posts

In this case, Combination 2 was the winner, despite being less specific about the reason Facebook is demoting Pages and posts.

Why was it the winner, then?

Because it tells a story – it describes an action. Combination 1, by comparison, describes a thing.

Active language can be a lot more compelling than static or passive language – and that’s something we’re going to explore a little more in the next section!

Which is…

Questions and answers

Which is more useful: asking a question, or giving an answer?

On the one hand, asking a question can make someone curious enough to want to find out the answer – but on the other hand, that’s asking them to do a certain kind of work they might prefer to avoid.

Here’s an example:

  1. [The MeetEdgar Newsletter] our remote workstations revealed 🤭
  2. [The MeetEdgar Newsletter] how does your home office compare? 👀

Between these two subject lines, Combination 1 was more popular – it promises an answer instead of a question.

(Our audience was MUCH more interested in seeing our home offices than they were in thinking about their own!)

Here’s another, which is a lot more subtle:

  1. [The MeetEdgar Newsletter] new data – how do you compare? 📈
  2. [The MeetEdgar Newsletter] new data – see how you compare 📈

These email newsletter subject lines are virtually indistinguishable, save for one small detail: the winner, Combination 2, offers the reader information, while Combination 1 asks them to perform analysis.

We’ve found over time that our newsletter subscribers consistently prefer when we offer information instead of giving them tasks, so this isn’t much of a surprise!

Here’s one more example of that – see if you can guess which was the winner:

  1. [The MeetEdgar Newsletter] upgrade your blog in literally 10 minutes ⏱
  2. [The MeetEdgar Newsletter] the 10-minute blog upgrade we swear by ⏱

If you guessed that Combination 2 won, you’re right!

Even though it seems like it breaks that pattern of action words being more compelling than static language, it goes to show how much patterns depend on context.

A subject line like “Facebook is demoting pages and posts” performs well for us because it tells a story about what someone else is already doing, but one like “upgrade your blog in literally 10 minutes” doesn’t, because it asks the reader to do work. In a case like this, a subject line teasing a static thing can be more valuable than one teasing an active verb – it all depends on who’s performing the action!

All of this illustrates how important it is to measure the success of your subject lines both in the context of your own readers AND the context of your subject lines over time.

You might find that certain rules only apply under certain circumstances – and that the things that work consistently for other marketers don’t necessarily work for you!

What have you learned from you own email subject line testing?

What are some of your subject line rules?

Do your readers find questions hard to resist?

Do they love emoji?

How do they compare to our own?

Let us know some of the subject line rules you try to follow in the comments below!

Social Share

Get Actionable Social Media Advice (And Not Too Much of It!)
Get The Dash, your weekly social media to-do list, delivered straight to your inbox.

Never Run Out
of Stuff to
Post on Social

MeetEdgar scheduling software automatically pulls posts from your content library to keep your social fresh everyday, on repeat.

  • Tom VanBuren

    Hey, we don’t take it personally! Truthfully, that sort of feedback is super useful – and I’m sure you’re probably right about not being alone in ignoring emails specifically marked as newsletters. (Just like including “newsletter” in the subject line might make other people MORE likely to open!)

    Who knows – maybe you’re inspiring one of the next things we’d like to revisit in a formal test…

  • Tom VanBuren

    One of my favorite tips is to approach them the same way we approach headlines – write a whole bunch of them, then whittle it down to your best choices!

  • Tom VanBuren

    These are excellent questions – I’ll do my best to address them!

    The reason we didn’t include more raw data is that raw data isn’t what this post is about – it’s about what happens after you’ve already collected and identified the data significant to you. In this case, that means, how do you unpack what makes one subject line more successful than another, as it pertains to language choices, tone, etc. When you’re looking at two subject lines that are particularly similar, how do you identify the differences in the messages they’re implicitly communicating, and can you identify patterns over time in the types of messages your audience prefers?

    Along those same lines, we could have gone REALLY in-depth and talked about analyzing open rates versus analyzing click rates, the balance of stoking curiosity while setting manageable expectations, and so on. We also could have made this even broader to talk about the difference between newsletter-type emails like these and sales/marketing emails! Ultimately, though, creating a sprawling, definitive guide to testing isn’t what we set out to do, so we stuck to including only the information most relevant to the points we wanted to make about the role of language choice and messaging.

    That said, I definitely understand what you mean – in theory, this could have been a significantly longer, broader, and more comprehensive guide to testing emails, identifying the data that’s statistically significant, and analyzing that data. And who knows, maybe that would have been even better – or maybe our title could have better reflected the narrower focus on language and syntax choices that this post ultimately has! We may very well find that you weren’t alone in your expectations of what exactly this post would cover. (Which brings us back to that question of “how specific is specific enough,” doesn’t it? Certainly this post covers what we learned from six months of testing subject lines – but it doesn’t cover EVERYTHING we learned from it.)

    As far as what defines a huge impact goes, I’d say that that’s fairly subjective – and like you said, it depends on whether you’re measuring size and significance in terms of percentage points or raw numbers. Because you’re absolutely right – from a percentage-based point of view, 1% isn’t a lot! But if a 1% increase represents 1000 readers we wouldn’t have had otherwise, we do consider it a significant gain, and one that can have a significant positive impact. Improvements that appear small on a percentage scale can translate into actual numbers of people that ARE huge – it depends entirely on your point of view, and what you consider to be a large victory. Which isn’t to say that we disagree with anyone whose point of view or priorities are different from our own – just that the ways people measure success can vary! If 1% doesn’t translate to what someone considers a significant change, we can absolutely understand and respect that.

    Sorry that got so long-winded – you brought up excellent points, and I wanted to show them the thoroughness they deserved! Your perspective on this was certainly appreciated!

  • Laura Mann Weed

    Great post! You made me feel pretty excited about my 41% open rate and 23% click through rate. LOL… love all these suggestions, no reason not to go higher!

    • Tom VanBuren

      We have in the past, and found that this version got a positive response – but that doesn’t mean it’s something we’ll never test again in the future, too!

Leave a Reply