AI Product Management: Research, Requirements and Scope
What is AI good at? What questions should you ask during the research phase? How should you prepare business requirements for the development team?
There are several ways companies become interested in AI:
- They talk to users, learn about problems, and gather requirements. And come to the conclusion that AI is the most suitable tool to solve the problem.
- Or they start research with questions such as “How can we use AI to our advantage?” or “How can this technology, everyone is talking about, help our users?”
In both cases, product managers need to answer baseline questions to define requirements.
Part 1. Problems and Opportunities
How Can AI Help Your Clients?
AI can solve a range of problems: optimising repetitive tasks, assisting with decisions and making decisions autonomously. If your users are facing some of the following problems, AI might be the right solution:
1. Is there time-consuming noncreative work that users do repetitively?
Every time IDEXX radiologists see a new patient, they spend around 30 seconds rearranging and grouping scans. It’s not only a waste of time but also an irritating task that specialists have to perform many times per day.
2. Are there tasks that need double-checking because of errors?
As I write this post, I’m relying on Grammarly and it’s AI algorithms to assist me.
3. Do your users spend a significant amount of time manually labelling data or creating complex rules for it?
Ataccama’s clients have hundreds of data sources and millions of tables. Usually, the data is unlabeled and hard to find. Data stewards spend a considerable amount of time manually labelling data and writing labelling rules.
Ataccama AI algorithms spot patterns in data, learn from the users’ actions and assist data stewards with labelling or automate it completely. It saves users from going through data sets one by one and doesn’t require writing rules.
4. Can assistance make users or the whole organisation more efficient?
In 2016, Moorfields Eye Hospital had 7K urgent referrals for people who were in danger of losing their sight. Patients had to wait up to 6 weeks before seeing a specialist because of the huge number of appointments. It turned out that only 800 of the 7K referrals were urgent.
DeepMind created an algorithm to assist ophthalmologists in analysing scans to diagnose urgent cases and reduce the number of deteriorations. Now it’s working at the level of world experts.
5. Are there extra tasks users have to complete to do their main work?
Data analysts spend up to 60% of their time searching and preparing data before they can start the actual analysis. It’s not only expensive for companies, but it’s also a boring task for analysts. AI can reduce these tasks using NLU and pattern recognition.
6. Does a user have to analyse the data and make decisions over and over again, many times per day?
In insurance, AI is used to calculate the price of damage based on images of a car. Trained using a library of photographs from past accidents, AI estimates repair costs. AI minimises the waiting time for payment and reduces work for insurance inspectors.
7. Do clients have rule-based systems?
AI would be overkill for simple rules, but if your users are creating complex rules with dozens of conditions that are changing over time, they would benefit from self-learning algorithms. Examples: Spam detection, segmentation, search, recommendations.
8. Are there systems they have to monitor and check regularly?
Machine learning is good at spotting patterns and anomalies in these patterns. If your users’ job is to make sure everything works (security, fraud, cyber-attacks, system failures, medical results), anomaly detection might help them spot anomalies faster and even find ones not visible to human eyes.
9. Do they have to customise work for their clients?
Customer support, marketing campaigns, chat-bots, reservation systems can be automated if there is enough information about customers and their past interactions with a service.
Zalando’s marketing content is generated by algorithms. They use information about past purchases and wish-lists of their customers to personalize marketing materials.
10. Is there a middleman between your system and end-users?
It’s the reality in a lot of large organisations. For example, to make business decisions, managers and strategists require access to data. Often they have to request data from their more tech-savvy colleagues in the IT department. It might take weeks or months until the data is available.
Allowing less-technical people to interact with the product in the natural language might change their work experience.
11. Are there problems and tasks that change over time?
Dynamic prices, navigation, logistics, supply chains and stocking are examples of problems that change over time. One of Ataccama’s clients is a large hospital group. They use data to predict how many ambulances they need at any given time and in a given location based on the time of day, week and year, weather, events, air pollution, traffic, and many other changing factors.
12. Is there information that might change your users’ business that is unavailable now?
Danny Lange (ex. Head of Machine Learning at Uber) recommends using the thought process “If we only knew ____” to uncover unexpected business possibilities. “If only we knew how many ambulances we need at any given time”, “If only we knew when to turn on the TV to see the most important moment of the game”.
Example from an Australian cricket broadcaster. Cricket matches can take as long as 5 days. To win, a teams has to take 20 wickets. About half a minute of meaningful moments out of 30 hours. If only fans knew when something interesting was going to happen and could be notified. Foxtel trained their AI algorithm, Monty, on historic videos of cricket and taught it to predict the chance of a wicket. Monty calls users to watch the wicket fall.
Is AI the Right Tool for the Problem?
Check-list to spot red flags in the beginning:
1. Do you have access to data? Can you acquire the data you need? Are users willing to share the data? What data are they willing to provide?
2. Do you have permission to use the data? Are there regulations you should contend with?
3. Can you ensure the users’ data is secure?
4. Can you ensure the data is trustworthy and up to date?
5. Is it worth it? Gathering data, keeping it in shape, building a model, and testing and iterating are both time and money consuming. Do you have a team? What is the trade-off? Maybe you need to update the model every day and it’s too expansive.
6. Can you solve the problems with simple rules and logic?
Part 2. Research
Once the problem is defined, it’s time for the research phase. The goal is to gather requirements and learn about constraints.
Answering the following questions will shape the scope and design:
1. Who are the users?
What is their expertise? Are they tech-savvy? What value are they expecting to derive from this product? How familiar are they with AI-based products?
2. Understand user attitudes towards data use
Are users willing to share the data? What data are they willing to provide and what data won’t they provide? Are there company- or industry-wide policies and regulations? What is considered ethical/unethical?
3. Identify the context in which users will use the product
Which environment does the problem occur in? What are they doing before and after? What type of tools do they use? Who do they interact with? What other needs or problems can occur at the same time and in the same environment?
4. What are other tools in their workflow?
If your product is part of a workflow, other tools may affect user habits and expectations. Are they using AI in their work? What type of AI? How are they used to interacting with other AI-based systems. Are there complementing tools you should take into consideration?
5. Who are the competitors?
Are they direct competitors? Would it be possible for your client to switch from one tool to another? How much money and effort would it cost?
6. What are the industry trends?
In the B2B and enterprise world, trends and analytics reports from research and advisory companies have considerable impact. What sources of information do they trust? Are they partnering with any of the advisory companies?
7. What part of the work are users proud of or would be reluctant to automate?
Automation of work can be a sensitive topic. How do we address users’ fears and explain the benefits of AI? Some work is tied to KPIs and bonuses. Your tool might automate or replace this work. How can you help clients to adapt not only to a new tool, but also to the new culture around it?
8. What level of automation should you aim for?
Should you aim for an autonomous system or collaboration between AI and a human? Some task users would outsource to AI, but there are activities people prefer to do to themselves. What are these tasks?
How much time and money would full automation save your clients? Is it worth it? What are the disadvantages of full automation?
📝 More about automation “Human in Control or Automate Everything?”
9. What level of accuracy is required?
How much would the error cost? In terms of money, time, reputation, health, delight, and experience. Are errors acceptable? What are the consequences of false-positive / negative results for the task? Would your clients rather have some errors in the prediction than manually solve tasks? The more accurate the results, the fewer predictions AI makes.
In the case where AI was used to calculate the price of damage based on images of a car, what percentage of the time could AI make the wrong estimations but still be profitable? How much would it cost if the AI needed the assistance of people every time it’s unsure?
10. How detailed do the results of interpretability/explainability need to be?
Is full interpretability required by law or policies? For instance, in banking, when a mortgage application is declined a client may request the exact reason.
There are many experiments using AI in radiology. To assist doctors, AI needs to provide all the data and explain the logic behind the prediction.
On the other hand, when Zillow estimates the market value for a property, it doesn’t provide all the data points or explain the algorithms they use. For the user, the explanation is short “We calculate the estimated range based on the current market and the info we have about this house”.
What information do your clients need to trust the prediction or make a decision based on AI suggestion?
📝 More about explainability “Explaining system intelligence Empower your users, but don’t overwhelm them”
11. How can you ensure the data flow?
The amount of training data is one of the most important components for building a precise system. When working on an AI solution, getting the initial training data and supporting continuous data flow is the product team’s job.
Research whether or not you can enrich your data with public data sets and get new insights. Do your clients have additional data you can use for their benefit?
12. How often should results be updated?
Can the prediction be calculated in advance or should it be updated every time the new information arrives (clicks, likes, photos, scans)? This question is important for technical requirements.
13. Pay attention to users’ communication style
How much jargon is appropriate, how much explanation do they need? Would your users rather have an assistant with a personality or would they trust numbers and percentages more? What terms are considered industry standard and what needs additional explanation? What words are used by experienced people and beginners?
Part 3. Business Requirements Check-list
The design of user flows and the interface depends on business requirements, scope, and a use case that a product management has prioritized. Both design and development teams need the following information to start working on a problem:
✓ Who will be using your product/feature?
✓ What problems do we want to solve?
✓ What type of impact are we aiming for (user satisfaction, reduce cost, minimise time, maximise safety)?
✓ What are the assumptions and hypotheses we want to test?
✓ What are the priorities?
✓ Use cases that are out of scope for this product/feature. Some use cases might be expensive to solve and would have a low impact on clients.
✓ Metrics (money, clicks, conversion rates, manual engagement rate).
✓ Data obtaining strategy and DQ metrics.
✓ User onboarding.
✓ Roadmap. What should the alpha-version look like? How will the feature evolve?
✓ How will the product be tested (monitoring system performance, adoption to change in data or user behaviour, data, model, usability)?
✓ How should we gather and work with user feedback?
✓ Known constraints (legal, data, trust)?
✓ What are the potential risks (biases in data, lack of data or trust, risks for reputation)?
✓ How do we ensure security and safety (from bad data, manipulation, theft)?
📝 More about business requirements for AI features Recommendation Product Driven Machine Learning (and NYC Parking Tickets)
Part 4. Interface Design
When requirements are defined, the design team can start working on the interface. Read about visualisation and prototyping in the next post.
https://medium.com/@nadyatsech/the-design-of-ai-based-products-13-things-to-consider-297ce9c0f0ba
Takeaways
I‘d say the number one thing product managers need to do is to align on the problem and important metrics for users. Changing the problem definition and the scope in ML development is much harder and may cause a lot of trouble. On the other hand, we can be less focused on details because the iterative nature of ML development supports changes and improvements.
Sources
📝 Articles
- Building AI-first products
- Product Driven Machine Learning. How business goals shape the AI building process.
- Machine Learning for Product Managers
- 3 Common Problems With Your Machine Learning Product and How to Fix Them
- The Step-By-Step PM Guide to Building Machine Learning Based Products
- How To Create A Successful Artificial Intelligence Strategy
▶ ️Videos
- How to Be a Good Machine Learning PM by Google Product Manager. What problems AI solves, solution examples, processes for managing AI projects.
- What Is Machine Learning for Product Managers Like by Google PM? Basics of AI for PMs
