Preference Testing on AppStoreResearch

Jonathan Kennedy
Nov 5, 2024

Preference tests are a powerful tool for validating design choices, marketing messages, and user flows before launching them to your entire user base. By collecting real user feedback, you can confidently move forward with changes that resonate with your audience, and avoid costly missteps. 

What can you test?

Feel free to test anything you like, from app page headlines to full app designs. 

Here are a few examples:

  • App page headline

  • Website headlines

  • General app layouts

  • Branding or logos

  • App onboarding screens

  • Menu designs

  • Button styles

  • Color schemes

  • Icon designs

  • Navigation styles

  • Feature prioritization

  • User interface (UI) elements

  • Notification and alert styles

  • Content hierarchy

  • Validate copy

  • Offers

  • User flows

  • App designs

  • Logos

  • Ads

  • Loading animations

  • Splash screens

  • Error messages

  • In-app purchase prompts

  • User engagement features

  • Gamification elements

  • Layout variations for different screen sizes

  • Personalization features

  • Social sharing options

  • Feedback mechanisms

  • And more

Targeting the right audience:

The key to great research is ensuring that you're getting feedback from the right people. Our platform allows you to reach a highly specific audience from our 2000+ Shopify and Shopify Plus merchants and operators database. This means your tests will be populated with professionals who have relevant expertise and experience.

Filters to help you find your perfect match:

- Business residence
- Role
- Shop category
- Shopify Plan
- Monthly revenue range
- Number of products for sale
- Size of team

How to run a preference test:


  1. Create your test: Set up your test by defining the questions, assets, or designs to be tested.

  2. Add images for testing: Include up to 5 images for participants to review.

  3. Add follow-up questions: Ask up to 5 follow-up questions to delve into participant preferences.

  4. Target participants: Select your audience based on specific criteria or use custom screener questions.

  5. Receive responses: Get feedback from well-matched participants within hours.

  6. Incentivize participation: Offer incentives, starting at $10, to encourage participation.

Paying for Preference testing

The minimum incentive is $10, and at least 10 participants are required.

The service fees are structured as follows: 30% for the Pay-as-you-go plan, 20% for the Plus plan, and 15% for the Pro plan.

Payment is required upfront.

Example of pricing

Suppose you want to offer participants $20 each for taking a test and you're aiming for 40 participants. If you're on a pay-as-you-go plan, the service fee is 30%.

Here's how the calculation works:

Incentive cost: $20 per participant x 40 participants = $800
Service fee: $800 x 30% = $240
Total cost: $800 + $240 = $1040

Thus, the total amount to pay is $1040.

Benefits:


  • Fast results: Receive participant matches quickly.

  • Customizable targeting: Ensure your test reaches the ideal audience.

  • Comprehensive testing: Test multiple images and questions.

  • Seamless experience: Manage the process from start to finish on one platform.

  • Expert insights: Gain actionable feedback from industry professionals.

Ready to leverage data for informed decision-making? Whether optimizing layouts or validating messaging, AppStoreResearch provides valuable insights quickly. Start preference testing today to enhance your app’s appeal and functionality.

[Create Preference test]


Preference test tips for researchers


1. Avoid mentioning incentives in the title

Keep your project title concise and professional. Incentive details can be in a different field elsewhere to maintain focus on the survey's purpose.

2. Provide a clear project description

Help participants understand the purpose of your Preference Test. Clearly explain what the project is about and what they can expect by participating. Keep it short and straightforward.

3. Keep questions simple and direct

Avoid overly complex or leading questions. Use clear language to ensure participants can easily understand and respond accurately.

4. Test one variable at a time.

Focus on a single element per test, such as a headline, button style, or color scheme. This ensures you can accurately determine what drives participant preferences.

5. Use high-quality visuals.

Ensure all images, designs, or assets you upload are clear and professional. Poor quality assets can confuse participants and skew results.

6. Limit the number of options.

Stick to no more than 5 options for testing. Too many choices can overwhelm participants and dilute meaningful results.

7. Define clear goals for your test.

Know exactly what you want to learn from the test. Whether it’s validating a design or understanding user preferences, having a clear objective will guide your decisions.

8. Avoid jargon.

Use language that is easy to understand, especially if your participants might not be familiar with technical terms.

9. Use follow-up questions wisely.

Ask follow-up questions that provide deeper insights, like the reasoning behind their preferences, but avoid overloading participants with too many.

10. Analyze results objectively.

Avoid confirmation bias by looking at the data objectively, even if it contradicts your expectations or initial assumptions.

11. Consider cultural and demographic differences.

When targeting a diverse audience, remember that cultural and demographic factors can influence preferences. Factor these into your analysis.

12. Be transparent about user data.

Let participants know how their feedback will be used. Being upfront builds trust and encourages honest responses.

13. Balance incentives appropriately.

Offer fair incentives to motivate participants, but avoid overly high rewards that could encourage rushed or insincere responses.

14. Repeat tests when needed.

If results are inconclusive or if preferences vary widely, consider refining your test and running it again with a larger or more targeted group.

15. Test regularly.

User preferences and trends change over time. Conduct preference tests periodically to ensure your designs and strategies stay relevant.

Menu

Menu

Menu