Bot or not: Measuring email metrics in a bot-inflated world

Measuring email opens after iOS 15 has been a challenge, and there are more challenges ahead as email clicks are rapidly inflated by email security bots. Although email remains a valuable channel in any media mix, reporting tools and attribution models must adapt to inflated email numbers. We dive deep into how our measurement team is navigating this new and changing world of inflated email numbers to deliver accurate reporting and actionable insights to clients.

Rise of email bots

The famous John Wanamaker quote, “Half the money I spend on advertising is wasted; the trouble is I don’t know which half” still applies. With his decidedly nondigital ads, Wanamaker didn’t know which ads led to which conversions. But, at the very least, he could be sure he was reaching only real people with his ads. Today we can track every impression, click, and interaction with our ads, but Wanamaker’s quote still rings true despite these advances. As much as digital makes it easier to track ad activity, digital makes it just as easy to fake ad activity using bots. Because of bots, we’re often unsure if our ads reach real people on some channels. For email, that has become especially obvious in the past few years.

iOS 15 was released on Sept. 20, 2021, and with it came a new email privacy feature that opens user emails inside a proxy server, whether a real user opened the email or not, leading to a spike in open rates. The email’s contents are then cached on a server and the cached version — which no longer relies on the tracked URLs from the original email — is delivered to the user. This cloaks real iOS email open activity behind a veil of bot opens. Although it’s possible to identify iOS opens, removing those numbers leads to underreporting on opens, whereas keeping the iOS opens leads to overreporting. Either way, the data refuse to tell the truth. Attempts have been made to project iOS opens, but any projections come with their own assumptions and potential inaccuracies.

Phishing emails killed the click rate

However, we can still rely on email clicks as an accurate gauge of email effectiveness, right? Yes, but not without deeper analysis. Although email bot clicks have been a low-level problem for some time, it could be handled with basic filters such as an invisible link that only bots would click. The problem rarely rose to a level that rendered the data meaningless. But, since March, we’ve observed a dramatic rise in bot clicks. Clickthrough rates above 10% and as high as 80% have become commonplace. The culprit? In response to a dramatic rise in malicious email attacks, many organizations have turned to email security software that clicks links in every email sent to an organization’s email server. The goal is to click on malicious links acting as a normal user but within a quarantined environment. If malicious links are found, then those emails can be removed before reaching an inbox. The side effect for marketers is that the email click numbers can no longer be trusted — at least not without heavy filtering.

Thankfully we have more tools and data points in our analytics toolbox to identify real email clicks than we do to identify real email opens. For instance, we can score a click on a bot scale using traffic source information and behavioral data on number of clicks by a device within a set timeframe. Versions of this approach are being tested and adopted across the industry. We have incorporated these approaches to our own bot filter algorithm and will be fine-tuning our algorithm as more data become available.

Changes to organic search

Citing privacy concerns, Google stopped passing organic search keyword information in referrer URLs in 2011. This shut off a valuable source of SEO data, leading many in SEO at that time to predict the death of the industry. Today organic search is still an important source of traffic and the role of the SEO specialist charged with optimizing that traffic has remained just as important. However, SEO has changed. The focus has shifted away from technical tricks and keyword metrics. The winners in organic search today focus on core marketing principles: foster trust with your audience and provide a great experience. These are principles that even Wanamaker, without knowing anything about the PageRank algorithm, hyperlinks, or even computers, would understand.

Email is dead, long live email!

Is email dead? Email measurement is more difficult than ever, but everyone still checks their email daily. Email is an integral part of everyone’s professional workflow and personal lives, including doctors. Just like organic search in 2011, email is far from dead, it’s just become much harder to measure. Although email open data are only directional and click data are harder to trust, it’s important to return to the core marketing principles and the key performance indicators (KPIs) that the business is ultimately trying to achieve.

A holistic approach

We take a holistic look at every report that goes out for our Deep Engagement products. This means zooming out and evaluating every data point to understand email in the context of the entire user journey. It also means zooming in with session recording software to gain insight beyond the metrics we track and to look at the interaction from the anecdotal view of the healthcare provider (HCP). It means constantly asking the question, “Are the data lying to us?” to validate anomalies in the data and uncover insights that could improve our products. Ultimately the goal isn’t to drive X number of email opens or X number of email clicks, but to engage HCPs and convey a brand message. Interpreting metrics and a user’s experience in the context of how effectively they drive your brand KPIs is a principle that will never die.

This article is written by Caleb Freeman, Director, Analytics at Healio Strategic Solutions.