Back to Community feed

Top A/B Testing Tools That Are Good for the Next 5 Years (Vetted by Privacy, Features, Maturity & Price)

Top A/B Testing Tools That Are Good for the Next 5 Years (Vetted by Privacy, Features, Maturity & Price)

 

So you’re looking to buy an A/B testing tool. Maybe it’s your first tool or maybe you’re in the market for a new one, but the key point is this:

You’re looking to get a tool that will serve your needs NOW, while also being able to grow with your needs and experimentation goals as they change over time so you won’t need to upgrade for a while after this.

That’s what we’re going to cover for you today.

Top Trends in A/B Testing & Why You Should Care When Choosing an A/B Testing Tool

1. A/B testing for Decision Making

Traditionally, CRO and A/B testing have been all about website optimization:

How do we get the traffic on our web pages to take more of the action that we want?

Nowadays, however, it feels like CRO and the entire landscape are evolving to a more holistic and strategic view. (Even the name CRO doesn’t quite fit with how it’s changing.)

Rather than just A/B testing software or thought process to measure conversions, marketing teams are now looking at it from the perspective of measuring and understanding all audience interactions to make larger business decisions.

CRO as a practice isn’t drastically changing, but its place in the organization has shifted.

What was once the domain of a sole team, or in some sad cases, a sole individual, is now an operating system by which data-driven teams across a company make decisions.

I’ve seen the shift from decentralized models to centralized teams, where all CRO efforts are owned and guarded by a core team of experts and specialists. Turns out, this doesn’t scale super well.

Now this model, in many mature organizations, is turned into a “center of excellence” model. In this way of operating, the CRO team simply enables and supports the rest of the organization’s experimentation and optimization efforts through education, process management, resource allotment, and product management (typically it’s at this stage that company’s invest in custom tooling or data pipelines to enable more sophisticated experimentation and greater scale).

Additionally, “CRO” as once practiced, is essentially called “web strategy” or “website product management” now.

Which makes a ton of sense, because growth product managers on the product side are doing the exact same work and calling it something different. This blending of the marketing and product side has yet to fully eclipse the previous silos, but when it does, it’s going to enable much more interesting full-funnel experimentation.

The tools that will help to enable this are the ones that balance server and client-side experimentation, have robust integrations, and have advanced features like experiment swim lanes as well as a mix of marketer-centric features (like your typical WYSIWYG editor) and product-centric features like feature flags.

Alex Birkett, BeOmniscient

2. A/B Testing That Minimizes Impact on Page Experience. 

The CWV update is all about improving the user experience on the page:

  • How fast it loads
  • How quickly the user can understand what the page is about (LCP)
  • How soon they can interact with the page
  • How quickly the page responds after that interaction request (TBT)
  • How much the page elements shift as the page loads or is used (LCS)

3. A/B Testing That Respects Privacy

A/B testing can be the best way to mitigate the risks of damaging the user experience in the future. If you think about it, testing is all about proving a hypothesis that one version is better than the others. So you really don’t need to store anything more than the number of users seeing these versions and the goals you’re tracking. Just that… two simple numbers, variationID and GoalID, and the number of times they are seen. You should set up your A/B testing in this clean way, nothing more.

You lose the trust of your users if you try to slice and dice your data in search of a winner. That’s not what A/B testing is for, that’s analytics — don’t mix it up.

Mitigate the risk of hurting your users and get an occasional win for the business. It will win you trust, save you money and so make you the best version of what you can be for your users.

If you keep it clean, you won’t be impacted by privacy laws. You don’t store any personal data and don’t track users. Your tracking usage… that is long-term the future of this niche in CRO.

Dennis van der Heijden, Convert.com

4.A/B Testing Powered by Data & Machine Learning. 

CRO is beginning to change and will very apparently change over the next 5 years, in two fundamental ways. CRO is becoming more empirical and data-sophisticated. (…)

In many ways, this is just a reflection of what the FAANG (Facebook, Amazon, Apple, Netflix and Google) companies have known and practiced for the last decade. Having realized the imperative of testing everything across the business and running as many experiments as technically feasible, FAANG companies have hired scientists to develop what are quite literally fundamental advancements to the field of statistics. These advancements allow them to run more experiments, run experiments faster, and measure increasingly complex outcomes (e.g. Bing measuring “did a user successfully find what they were searching for?“, or Netflix measuring how different approaches to loading videos on slow connections impacts user frustration).

The areas where CRO was previously thought of as more of an “art form” (guiding program strategy, roadmap prioritization) will increasingly rely on science and machines that can make more optimal, and empirical decisions about how to prioritize potential treatments to test and what audiences to test on.

Ryan Lucht, Senior Growth Strategist, Cro Metrics

Top 5 A/B Testing Tools That Are Good for the Next 5 Years (Based On Current CRO Trends).

#1: Convert Experiences

Convert Experiences

G2 Rating: 4.7/5.0 (46 Reviews)

Pricing: Starting as low as $699 per month, with $199 for every 100k visitors after that.

Do they offer a free plan or trial? Yes, 14 days free trial with no credit card needed.

Cost per 100,000 visitors: $199.

Pros:

  • Fully privacy compliant: no personal data is ever stored
  • Fast and flicker-free
  • Feature-rich, run unlimited tests
  • Integrates with 100+ 3rd party tools (e.g., Shopify, WordPress, Mixpanel, Hotjar)
  • Reliable, fast customer support.

Cons:

  • Basic post segmentation on the Kickstart plan

Client Side, Server Side or Both? Convert works as a client-side WYSIWYG editor, but it can also be set up on the server side and run custom JS.

Frequentist or Bayesian? Convert Experiences runs experiments using the Frequentists Two-Tailed Z Tests.

Does it support Full Stack? Currently being implemented. Join the waitlist here.

Core Web Vitals Ready? Yes. The Convert script loads incredibly fast, is flicker-free, and has minimal effect on CLS.

Has ML or AI? No.

Do they serve Enterprise? Yes. We are the tool of choice for companies like Sony, Jabra, Unicef, and others.

Do they offer Customer support? Yes, right from the start of the trial. What type? Live chat, blog, and knowledge base with more to come.

Do they care about Privacy? We are fully GDPR compliant. We even stopped using tools for our internal work from other companies that don’t meet GDPR, that’s how much it means to us.

Do they care about the World? Yes. It’s in our DNA. We plant trees, run community programs, champion diversity from the initial application, donate to charities, and much more. Heck, we’re 15x carbon negative.

Trust Radius Review:

 

See what Convert Experiences looks like in action.

Experience privacy-compliant, flicker-free, limitless testing. Try Convert Experiences for free for 15 days.

#2: Google Optimize/ Optimize 360

A/B testing tools Google Optimize

Google Optimize

G2 Rating: 4.3/5.0 (27 Reviews)

Pricing: There are 2 versions of this tool. Optimize is free, but for advanced features, you have to opt for Google Optimize 360.

The pricing for Optimize 360 is custom (although rumored to be around $150,000 per year). To get your monthly price, you’ll have to fill a form to contact their sales team.

Do they offer a free plan? Yes.

Cost per 100,000 visitors: N/A

Pros:

  • User-friendly — little technical knowledge required
  • Great for first-time A/B testers
  • Quick integration with Google Analytics
  • Has a super detailed audience segmentation, thanks to Google’s huge data resources

Cons:

  • Limited to 5 tests at a time on the free tier
  • No drag-and-drop editor functionality
  • Cannot upload your images directly
  • Cannot test apps, only browsers
  • Not suited for complicated tests
  • UI/UX and reports aren’t as visually appealing as most other tools
  • Flicker happens sometimes.

Client Side, Server Side or Both? Both.

Frequentist or Bayesian? Google Optimize uses Bayesian methods rather than Frequentist methods, also known as Null Hypothesis Significance Testing (NHST)

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool loads fast, although it can flicker which might cause CLS issues.

Has ML or AI? No.

Do they serve Enterprise? Yes, but only via Optimize 360.

Do they offer Customer support? They have a resource hub with tips, video tutorials, help community, and more.

Do they care about Privacy? They comply with applicable data protection laws in the countries where their products are used.

They also work with third parties and data protection authorities to keep users’ data safe.

Do they care about the World? Google has been carbon neutral since 2007 and plans to be carbon-free by 2030. Apart from that, they’re known for numerous philanthropic gestures with their $1 billion commitment.

Trust Radius Review:

 

What this A/B testing tool looks like in action:

 

Need a tool that ranks better than Google Optimize for several critical factors like “Quality of Support”, “Ease of Doing Business” and “A/B Testing”? Here’s how Convert Experiences stacks up against Google Optimize & Optimize 360.

#3: AB Tasty

A/B testing tools A/B Tasty

AB Tasty

G2 Rating: 4.5/5.0 (76 Reviews)

Pricing: Starts at $1900/mo for 400k monthly tested visitors for the Essentials plan. The Growth plan costs $3800/mo while the Elite plan starts at $5700/mo.

Do they offer a free plan or trial? No. You can request a demo call to see their new features.

Cost per 100,000 visitors: Approx $475.

Pros:

  • AI and ML built into the optimization
  • Run unlimited experiments
  • Clean and easy-to-use user interface
  • Easy to set up and preview tests
  • Dynamic Widgets
  • Multiple integrations
  • Extensive analytics reports
  • Wide range of targeting options available, along with personalization
  • Reliable customer support
  • Pricing is mid-range

Cons:

  • No automation to give insights about audience performance (especially for an active experiment) and will sometimes require customer support to see
  • Statistical Significance calculator is a bit basic in its UX
  • Google Analytics integration is complex, it requires coding
  • No free trial

Client Side, Server Side or Both? Both

Frequentist or Bayesian? Bayesian.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool loads fast.

Has ML or AI? Yes.

Do they serve Enterprise? Yes. They are the tool of choice for companies like Disney, L’Oreal, Kalviyo, and others.

Do they offer Customer support? Yes. They have a knowledge base and live chat.

Do they care about Privacy? Yes. User IP addresses are used to create an ID code and then immediately deleted before moving into the tool. ID codes are anonymized and then deleted after 13 months.

Do they care about the World? They donate directly to NGOs, they work with social action groups, they recycle and sponsor beehives.

Trust Radius Review:

 

What this split testing tool looks like in action:

 

Need an AB Tasty alternative that (actually) aces A/B testing? Take a look at the alternative that beats AB Tasty pricing.

#4: Optimizely

A/B testing tools Optimizely pricing

Optimizely

G2 Rating: 4.3/5.0 (109 Reviews)

Pricing: They’re using a custom pricing model. But Splitbase predicts they cost at least $36,000 per year.

Do they offer a free plan? No. They stopped the free plan in 2018.

Cost per 100,000 visitors: N/A

Pros:

  • Run unlimited experiments
  • Clean and easy-to-use user interface
  • The widget feature is fun to use
  • Wide range of targeting options available
  • Reliable customer support

Cons:

  • Doesn’t give automatic insights about audience performance (especially for an active experiment)
  • Google Analytics integration is complex, requires coding
  • Optimizely snippet usually increases page loading time

Client Side, Server Side or Both? Both. Optimizely offers client-side experimentation through a Javascript snippet and server-side experimentation through developer SDKs.

Frequentist or Bayesian? Optimizely’s Stats Engine uses sequential experimentation, not the fixed-horizon experiments that you would see in other platforms.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes, this tool loads fast.

Has ML or AI? Yes.

Do they serve Enterprise? Due to pricing, it mostly caters to an enterprise level. Brands like Microsoft, IBM, HP, eBay, Yamaha, Pizza Hut, and Atlassian use it.

Do they offer Customer support? Yes. They have a bank of resources to help users get unstuck and phone numbers to call for help 24/7.

Do they care about Privacy? They take into consideration old and new privacy laws and integrate that into their products so that you, the user, don’t have to worry about compliance.

Do they care about the World? Most new hires are sent to volunteer in the community on their second day.

Trust Radius Review:

 

What this A/B testing tool looks like in action:

 

Turned off by Optimizely pricing? Meet Convert Experiences! It’s the best of both worlds – it has all your favorite A/B testing features with 4x faster support that will save you up to USD 100k on a contract.

#5: SiteSpect

A/B testing tools SiteSpect

SiteSpect

G2 Rating: 4.4/5.0 (50 Reviews)

Pricing: You’ll have to contact SiteSpect for a customized price according to your needs.

Do they offer a free plan? Yes, they offer a free trial.

Cost per 100,000 visitors: N/A

Pros:

  • Supports all markup languages (HTML, WML, XML, and JSON), style sheets, and scripting languages
  • No javascript tag means no content refresh or flicker
  • Versatile enough to test almost any scenario
  • Non-intrusive testing
  • Integrates with analytics tools
  • No need to modify the production version of your code

Cons:

  • Technical knowledge required to implement tests
  • Adds an extra hop that could slow down your site a bit
  • The reporting interface could be better

Client Side, Server Side or Both? Both.

Frequentist or Bayesian? SiteSpect uses a two-tailed t-test when comparing Variations against Controls to determine the point at which confidence intervals do not overlap and thus indicate significance. SiteSpect also computes the z-score and uses it in reports.

Does it support Full Stack? Yes.

Core Web Vitals Ready? Yes. This tool is incredibly fast.

Has ML or AI? Yes.

Do they serve Enterprise? Yes. Staples, PetSmart, AmericanGirl, and Urban Outfitters use SiteSpect.

Do they offer Customer support? Customer support is available via phone call and email for free and paid versions. There are also pre-recorded webinars available in their knowledge base.

Do they care about Privacy? They do. They are PCI DSS 3.2 certified; GDPR, CCPA, and privacy shield compliant, and HIPAA ready.

Do they care about the World? SiteSpect is known to sponsor some charity projects since 2014.

Trust Radius Review:

 

What this A/B testing tool looks like in action:

 

Read the original article here: https://www.convert.com/blog/a-b-testing/a-b-testing-tools-2022-beyond/

Articles — Other
ActiveCampaign
Sponsored
ActiveCampaign is a marketing and sales automation platform that helps growing businesses meaningfully connect and engage with their customers. Its SaaS platform enables businesses to create optimized customer experiences by automating many behind the scenes processes and communicating with their customers across channels with personalized messages.
0 answers, 3 articles
Reply
0 answers, 0 articles
Reply