Increasing email sends with industry benchmarking insights
Desired outcome
Empower our customers with industry benchmarking data to improve their email marketing campaign strategies
Product Manager
2 Frontend Engineers
2 Backend Engineers
1 Content strategist
Impact
Statistically significant feature adoption and increase in email campaign sends post-launch
The Problem
Through customer interviews, we learned that our customers did not feel like they had a clear understanding of how their marketing campaign performances compared to that of their peers. Additionally, quarterly survey data indicated that the 2nd top most feature requested by customers was a way to compare audience performance data. Based on this data, our team, the Audience Management team, was asked to prioritize this feature request.
The Approach
Conceptualizing our approach
I designed a quick low-mid fidelity concept design of the feature in isolation that showcased all of the data points we planned on using together in one interface. We interviewed users and took in early feedback on the concept design artifact. We received enough validating feedback to move forward with higher-fidelity designs and prototyping with minor customer feedback being incorporated into the followup approaches.
Our team got together to determine how to build an initial version of an audience comparison feature. We knew that we had existing data that we could leverage to help us make such a comparison. We needed to understand what data to compare, how to identify “peers” or campaign data that had similar characteristics, and then we had to determine what characteristics of a business we had available to use in our comparison. Lastly, we needed to determine where we would surface this feature.
Using predicted audience demographics
Another key point of data that we were able to leverage was a feature that our team worked on earlier in the year, predicted audience demographics. We would use this data point to help us make a comparison between other email campaigns with similar audiences, specifically audience size and audience gender.
Using the customer's business vertical
When a new customer was going through onboarding, they were given the option to self-categorize the business vertical that best matched their organization. We could use this data to help paint a picture of similar businesses within the same business vertical when doing our comparison. In collaboration with our Data Science team, we trained a ML model to analyze thousands of email campaigns to determine the business vertical. We used this ML model to predict the customers’ business vertical In the event where a customer hadn’t selected one during onboarding. We also provided a way to update their business vertical if the prediction was incorrect.
Exploring data visualization
While open, click, and unsubscribe rates existed for the individual campaign report itself, we didn’t have data visualization that showed the comparison between their peer data. I tapped into our design system’s chart patterns and iterated through a few different ways that we could visualize the comparison between the email campaign’s performance against the average metric performance of their peers. After several team and internal stakeholder reviews, including reviews with our data scientists, I landed on a simple bar chart pattern to visualize the comparison data.
Reccommendations
We enhanced our feature with a recommendation system that provided guidance based on the results of the campaign. For instance, if the campaign remains unopened, our recommendation would be to re-send to those un-opened contacts. When the campaign performs above industry averages, we surface a congratulatory message.
Outlining the user journey
With the essential components and elements selected, we were ready to put concept to code. I outlined the user journey for the experience using the following user flow. This would illustrate the "happy path" the user would take as well as capturing edge cases along the way, for example, when to show a celebration moment for above average performance and when to show a banner suitable for underperforming campaigns.
Testing in the wild
Now that we were moving forward with our concept, we had moved into higher fidelity designs and wanted to get additional user feedback on our more high-fidelity approach. To do this, our team took advantage of an opportunity to visit small business owners who were Mailchimp customers at their various places of business onsite in Portland, OR. We spent time with 5 Mailchimp customers who were also primarily in charge of handling their email marketing for their business. We also came away from the trip with a few observations:
Wishlist feature: Self-comparison
While participants saw value in the feature of comparison data against their peers, almost every participant mentioned a desire to compare campaign performance data against their own prior marketing campaigns
Trusting & recommendations
While the comparison data was valuable for participants, some expressed a desire to see some sort of proof that the recommendations provided would actually have the desired impact. In other words, they wanted to see that recommended action work for someone else before trying it themselves. This ended up being a major insight for us regarding recommendations in general.