Experiments
Last updated: October 7, 2025
Overview
At FareHarbor, we’re committed to continually improving our products and services so that both our clients and their customers have the best possible experience. One of the ways we can validate or identify opportunities for improvement is through experiments – short-term, controlled tests that help us learn which changes make the biggest impact.
When designing an experiment, we are intentional about what we test and set clear eligibility criteria for participation. Our top priority is to ensure that every experiment is conducted with safeguards to protect our clients, minimizing disruptions to daily operations and negative impact on revenue.
What is an experiment?
Experiments, such as A/B tests, are short-term, controlled studies that compare different versions of a feature to see which performs best. Not every experiment leads to a general release. Sometimes, the data shows an idea needs more refinement or doesn’t provide enough value to pursue further. Either way, experiments help us make informed decisions about where to focus our efforts for the greatest impact.
Examples of experiments
- Copy updates: Testing different copy for buttons, labels, or instructions
- Layout adjustments: Reordering or resizing elements on the page
- Design variations: Trying alternate colors, fonts, or icon styles
- Behavior changes: Modifying how steps or fields appear during checkout
- Call-to-action tests: Experimenting with placement or emphasis of booking buttons
- Pricing displays: Testing how price elements such as fees, taxes, and other costs are displayed
How do experiments differ from beta testing?
While both help us learn about new features, they serve different purposes:
- Experiments are short-term, controlled studies (often A/B tests) that measure how variations of a product or feature affect behavior. They typically run quietly in the background and answer questions like, “Does this change make the booking flow clearer?” or “Does this button copy improve conversions?”
- Beta testing gives a group of clients early access to a new feature before it’s widely released. Beta testers use the feature in their daily operations and provide feedback on functionality, reliability (bug identification), and overall value.
Finding information about experiments
Admins at FareHarbor can find information about ongoing experiments in #experiments-updates. Each record in the Experiment Updates Airtable includes a link to its kick-off post, along with a list of the team members responsible.
If a thread for the experiment you’re looking for doesn’t exist, reach out to the Product Enablement Manager (PEM) listed in the experiment’s Airtable record.
Note: To keep information clear and consolidated, we kindly ask that you post in the relevant thread rather than directly in the channel!
Why it’s important to keep clients in experiments
Including clients in experiments is essential to understanding how real-world users interact with our product and features. Experiments provide accurate, data-driven insights that can’t be replicated in internal testing alone. By observing end user or client behavior in live environments, we can identify what truly improves clarity, usability, and conversion rates.
Clients are carefully selected for experiments based on requirements needed to achieve statistically significant outcomes. While they may not always see the purpose of a test, keeping them enrolled ensures we capture a full range of feedback. Both positive and negative responses are valuable — and knowing whether the audience adopts or ignores a feature provides equally important insight. Opting clients out reduces the quality and reliability of the insights we gather, since every action (or lack thereof) contributes to understanding overall behavior.
How to talk to clients about experiments
As a general rule of thumb, we don’t proactively communicate experiments with clients. This is standard practice across SaaS companies, since experiments are short-term, limited in scope, and may never become permanent features. Communicating every experiment could create unnecessary confusion or raise expectations for changes that may not be released.
Experiments are also intentionally designed so they do not require client approval. They typically focus on small adjustments (see examples) that allow us to gather insights without disrupting operations.
Reactive communication
If a client reaches out about an ongoing experiment, use the following language to respond. This messaging highlights the value of experiments and reassures clients that any changes they notice are intentional and part of our continuous improvement process.
- Close template: Product – Experiments – Reactive Comms
- Zendesk macro: Experiments – Reactive Response
Opting out clients from experiments
We generally discourage opting clients out of experiments. As noted, including a broad range of clients is essential for collecting accurate, representative insights.
Steps to take before escalation
- Highlight the value of experiments and reassure the client that experiments are designed to be low-risk and non-disruptive.
- Use the reactive communication prepared to explain that the changes are intentional and part of our improvement process.
Escalation process
If a client continues to raise significant concerns, review the following criteria before submitting an opt-out request:
- Client is tier 2+
- Negatively impacting or disrupting their business performance or operations.
- Misrepresenting their brand, services, or user experience in a way that could lead to confusion or harm.
If the above criteria are met, submit an experiment opt-out request form. Requests based solely on a client’s preference or discomfort with change will not qualify for removal. If the client is escalated, please reach out to the Product Enablement Manager (PEM) listed in the experiment’s Airtable record.