Built for Consumers
2024
Project Overview
Consumer Reports, a non-profit consumer advocacy group founded as Consumers Union in 1936, has provided consumers with published product reviews based on rigorous, transparent and scientific testing methods. As the marketplace for consumer products shifted considerably from simple physical consumer goods to complex digital product service ecosystems, they have encountered new challenges in protecting and empowering consumers in the digital age.
To face this challenge, Consumer Reports created an Innovation Lab, which has recently developed a variety of new digital products to tackle these emerging challenges. Some of these products include, but are not limited to, Permission Slip, Security Planner and Upkept. Seeking to keep pace with the speed of the emerging Artificial Intelligence boom, they sought design inspiration on AskCR, an authorized intelligent agent that was being developed by Consumer Reports for release to subscribers and the general public.
The Problem
In the emerging age of Artificial Intelligence even more problems and possibilities lurk behind the corner of innovation. As such, the Consumer Reports Innovation Lab posed us with the following challenge:
Design an intelligent agent, system and/or platform that ensures that companies deliver on their promises by mediating interactions between individual consumers and companies regarding warranties, privacy policies, terms of use and other intangibles.
The Process: Highlights
To see the full process book for the project, please click here to view an electronic copy available for download.
Abstraction Laddering and Co-Design
Some of our initial activities in helping us understand the problem space included our kick-off meeting with Consumer Reports Innovation Lab, where we traveled to their New York headquarters and walking through some design activities, including an Abstraction Ladder. This helped us generate potential ideas to explore later in our design process and helped us empathize with our client and understand their perspective on the problem.
Customer Journey Mapping
Following the kick-off, we put together a Customer Journey Map for general consumers of both physical goods and software products and services. Given Consumer Reports increasing focus in the digital space and residual brand equity in physical goods, we wanted to include both journeys early in our research and design process.
Customer Journey Map of Physical Goods
Customer Journey Map of Software Products
Pretotyping
At this phase, we knew roughly what the client wanted, but not how. In order to invite failure into our process earlier, we began with paper prototypes ("pretotypes") to test our riskiest assumptions. Generally, we found that people were willing to let AI act on their behalf in low-stakes scenarios.
Storyboarding for Structured Interviews
Following the insights derived from the early paper prototypes, we then expanded on our sketches by storyboarding potential customer service interactions with a smart agent. We covered an array of industries and consumer products in our storyboard tests, including airlines, internet services, loan refinancing and car insurance. Our goal was to further understand what industries and risks consumers were most comfortable with when it came to a smart agent acting on their behalf, using these storyboards to guide structured interviews.
Synthesis of Qualitative Data
Our initial structured interviews yielded a high volume of qualitative feedback related to our goal of understanding desirability of smart agents acting on their behalf and the factors that influenced their risk tolerance.
This yielded an insight that allowed us to summarize a potential opportunity space for our project: People are more willing to let AI act on their behalf when the rewards for action are relatively high and where the risk of personal harm through disclosure of PII or product errors were minimal. Below is an Impact/Difficulty Matrix and Affinity Diagram that helped us organize our qualitative findings.
Synthesis of Quantitative Data
Throughout our research process, our team made use of a variety of quantitative market data that was provided to us by Consumer Reports. This secondary research data provided insights on the current state of the consumer experience. Drawing insights from this quantitative data, in tandem with the qualitative insights that we had generated through our own research, we were able to identify more use cases and potential domains of interest for our prototyping. Included are some images of the type of quantitative data that informed our prototyping decisions.
Use Case, User Story and Personas
By combining our qualitative and quantitative data from our primary and secondary research activities, we then outlined some common use cases to help us build more interactive and realistic prototypes. Additionally, we were able to address multiple user segments in our prototype development process through the generation of several user personas that addressed key user groups.
Research Through Design: Parallel Prototyping
After receiving client feedback, we began to converge on some key concepts. In order to test a variety or assumptions, we decided to parallel prototype. The result was three distinct prototypes that we were able to take into the next iteration of testing. The three prototypes were:
Negotiation Helper
Negotiation Helper is an AI-powered conversational agent that listens into users customer service calls, texting them information, guidance and tips in real-time during the call.
Policy Assistant
Policy Assistant is another smart agent that takes in a repository of company policies, terms of service and regulatory measures. It helps clients take advantage of rebates and refunds, notifying users of these opportunities and guiding them into taking advantage of them.
CR Wallet
CR Wallet is a do-it-all, AI-powered augmented reality concierge. It tracks your purchase using other existing Consumer Reports products and services, then meets you in the moment to help inform and guide users from the unboxing phase all the way to service request.
Research Through Design: Prototype Testing
These three prototypes tested different types of action and intervention from the smart agent. In order to provide a taxonomy to these levels of action, we coined the phrases "Inform, Guide, Act" in order see how comfortable users would be being informed by a smart agent, guided by a smart agent and finally, letting the agent act on their behalf.
Negotiation Helper tested the informing and guiding levels of action. Policy Assistant did the same. Meanwhile, CR Wallet went the whole mile, informing, guiding and acting on users behalf to accomplish customer service related goals.
Our prototype testing in this phase helped uncover some useful features that found their way into our final deliverable. Specifically, the company phonebook and pre-call tips were uncovered in this early testing as important features.
The Process: Product Strategy
Integration Into Existing Consumer Reports Product Service Ecosystem
After concluding our research phase, we examined the existing Consumer Reports product portfolio and brainstormed potential ways that we could add or generate value through the addition of our product. Their rapid innovation between 2020 and 2024 was crucial in determining how we could complement their existing products and services.
Branding
Through survey methods and A/B testing, we were able to come up with a unique brand that fit the Consumer Reports identity and captured the spirit of our product: Fairplay.
Adding Value for the Consumer Reports Product-Service Ecosystem
At this stage, we looked at the Consumer Reports product-service ecosystem to determine if our product could be complementary to any existing products or services. For example, Consumer Reports offers a B2B product to businesses that contains a wealth of consumer insights from their own subscriber base and general research. This product is called CR Data Intelligence, and it has emerged as a successful product that yields a significant amount of revenue for Consumer Reports. But how could we generate even more value for this product or others within the Consumer Reports portfolio?
Co-Creating Value for Businesses and Consumers
From interviews with both prospective users as well as business stakeholders, we derived the following Value Flow Map for our product and service. Given that our market was two-sided, based on Service Design principles, we knew that we had to create a product-service ecosystem that co-created value for both businesses and users. Additionally, this opened up more avenues for monetization by providing value for both stakeholder groups. For example, we designed the service to enhance the value to businesses who pay for CR Data Intelligence, opening up an additional and more stable revenue flow for Consumer Reports. Through creating and adding data to CR Data Intelligence, Consumer Reports could easily increase revenue through their B2B product without cannibalizing revenue from their more diverse consumer product portfolio.
Product Pricing and Customer Lifetime Value
Through our own survey with prospective users that fit our data-driven personas, we probed them on how much they'd be willing to pay for specific features and functions of our prototypes. Based on this, we were able to determine what our most desirable features were, and began constructing what a market demand curve might look like. On average, test subjects that we surveyed were willing to pay $26.50 annually for an AI agent that handled customer service issues on their behalf. From this analysis, we came up with the following proposed pricing structure in order to realize revenue from paid consumers. We also advised the use of metered usage as alternative monetization model during text-based interactions.
In order to grow subscribers to Consumer Reports, which comprises over 90% of their revenue, we included the product within the full bundle of services that paying subscribers get with their CR subscription. The goal of this decision was to add new subscribers overall as well as improve the value of the CR subscription to current subscribers so as to reduce subscriber attrition rate. Both of these goals sought to improve the user lifetime value to Consumer Reports and the Fairplay app.
The Process: Development Phase
Converging on a 1.0 Prototype
Following a Co-Design session with Consumer Reports Innovation Lab, we had a list of features and functions that our clients prioritized most. Now came the challenging part, which was balancing the interests of the various stakeholders on the project.
In order to ensure that we did not waiver from serving the consumer first, we looked at our existing research and this new client feedback and combined their respective strengths to develop one premium prototype.
Iterative Prototype Testing
After extensive structured interviews on this new combined prototype, we then engaged in subsequent rounds of designing prototypes and testing them. Our most important insights came with our 2.0 prototype. In that round of prototyping, we identified several additional core features: Real-time text-based tips for customer service chat interactions and using LLM's to help consumers generate arguments and build evidence-based cases to empower consumers to exercise their rights.
Iterative Development and Testing
As the result of multiple rounds of testing, we further developed our understanding of how users interacted with our AI agent. Generally, people felt uncomfortable listening into the conversation and supervising the agent, but still saw value and were willing to pay for an AI customer service agent to act, negotiate and advocate on their behalves. As such, we used design concepts such as feedforward that would provide the user with control over what the AI agent could and could not do for them during calls and text conversations.
Error Recovery
Given the agency that our product was designed to have and the potential consequences to users in the event of an AI error, we wanted to test methods of error recovery. Additionally, consequence scanning was a key focus of our interviews and was pivotal in terms of influencing how much the agent could do on behalf of the user and how the user could discover, explain and repair the error and its associated consequences. We also designed the product to take a minimalist approach to personally identifiable information gathering.
User Control and Personalization
Through our research and application of basic Human-AI Interaction principles, we built a high degree of user control into the product. This allowed the user to vary the amount of agency that our product had in decision making or the modality of their interaction with the company. We also included features to improve the user experience, such as customizing the agents tone during voice interactions and making the product's interfaces and recommendations more personalized for each individual user.
Interaction Design
Through testing various scenarios involving error recovery, AI agent behaviors and modality of Human-AI interaction, we refined our product's interaction design to accommodate human factors and lower user error rates.
Technical and System Design Considerations
At a high-level, we envisioned our system of agents to interact with each other through the following diagram. Should there be scaling issues, our usage of LLMs could be consolidated to reduce needs for processing power through the tailored design of Small Language Models should the Consumer Reports engineering team be cost-constrained with respect to processing power.
The Results
Final Product Description
Fairplay is a system of AI models that interact to create one holistic AI Agent for consumers trying to resolve disputes with product or service companies. It can support customer disputes with companies involving any good or service. The product relies on existing bank of data that Consumer Reports has already gathered over the course of their operations on consumer behavior, optimal company support channels and product or company policies such as, but not limited to: Warranties, Rebates, Returns, Recalls or Benefits. Through using the product and associated agentic services, users can reduce cognitive load, emotional strain, time investment and financial costs associated with resolving disputes with businesses and existing customer support channels. Our product meets users in the moment throughout the purchase and post-purchase phases of their customer journey for any product or service. Simultaneously, by feeding data into Consumer Reports' CR Data Intelligence product, we could also improve the value of the data services that this B2B product provides to partnering businesses, thereby leveraging synergies to expand the revenue streams from CR Data Intelligence and other Consumer Reports products, such as Billshark.
Omni-Channel Support
Through our development testing and market research, we realized that customers needed to be supported across different communication channels in order to mediate disputes with their product or service companies. As such, we developed a final product with a variety of features to meet user needs in the moment, whether it is through iOS iMessage plug-ins, voice interactions on calls or real-time policy tips.
Other MVP Features
Aside from supporting all channels of customer support through our agent's voice and text-based interactions, we also included the following features that were extrapolated from the findings in our prototyping and research phases.
MVP and Product Roadmap
Given our research and our consideration of the engineering and product development work already done on the AskCR product, we came up with the following proposed roadmap to continue our work and integrate it into AskCR's roadmap. We considered the AskCR roadmap given Consumer Reports' intention to expand its use case from the pre-purchase phase into the purchase and post-purchase portions of the customer journey.
Video Demos
Included are some video demos of our product and some sample use cases.
Meeting Client Expectations
So far, some major accomplishments for us was Consumer Reports renewing their Capstone contract for CMU in advance, as a result of the strong impression that we left. Another key accomplishment was that, based on our co-design session and work in the first half of the project, Consumer Reports volunteered software engineers and developers to help us build out our final deliverable. This was not a commitment that they made within the scope of the original contract, demonstrating the value that they see in our work.
Lastly, they published all of our work as part of their official Innovation Lab blog series.