Evaluating the Usability of Apple CarPlay
with Xueyin Liu (Airy) & Pengfai Cai
Launched in 2016, Apple CarPlay is an in-vehicle infotainment system that incorporates iPhone’s features in a car’s built-in display providing a smarter and safer way for drivers to use these features on the move. With this study, we wanted to understand how new and experienced users make use of the interface in conjunction with Siri and what are the usability issues they face while doing so.
Recruitment and User Demographics
We created a Recruitment Questionnaire and sent out the survey to different student-led Facebook and Whatsapp groups. We wanted to make sure the user:
- Is at least 18 years old,
- Is an iPhone user, and
- has a valid driving license
In total, we recruited 5 users ( 3 new — U1, U2, U3 — and 2 experienced — U4, U5 — ) who met the eligibility criteria.
Usability Test Details
Background research helped us identify Maps, Phone, Music, and Messaging as the four most commonly used functions on car infotainment systems. Therefore, we decided to test out these particular functions for our usability study.
Duration
30–40 minutes per session
Approach
We used the Think-Aloud technique along with direct observation to evaluate the usability of Apple CarPlay. In addition, we video-recorded the whole session to further analyze any issues that we might have missed out on during the live test.
Equipment
Volkswagen Jetta (2019) with Apple CarPlay installed
Methodology
- Introduction and Verbal Consent: We started out our tests by introducing the participants to the study and taking verbal consent to audio and video record the whole test.
- Pre-Test Interview: This was followed by a pre-test interview where we asked users questions about their past experiences with infotainment systems, the features they made use of and the problems they faced while doing so.
- Scenario-based Usability Test: This was followed by our scenario-based usability test where we asked the users to perform a set of tasks based on a given scenario. After each task, we asked the participant to self-measure the difficulty of a task on a 5-point Likert scale ranging from very easy to very difficult.
- Post-Test Questionnaire (SUS): After all the tasks were performed, we asked the users for final feedback and asked them to fill out 10 system usability related questions based on the SUS Questionnaire.
Scenarios and Tasks
We formed a total of 10 tasks — 2 to 3 tasks for each of the functions we were testing i.e Maps, Phone, Messaging, and Music. In order to make the test more realistic and to better engage participants, we made 5 scenarios in a continuous narration to incorporate these tasks.
Scenario 1
This is the car you just bought. And your friend Joe has just moved into his new house. You want to drive your new car to visit his place for a housewarming party. You want to set up your Apple CarPlay before departure.
Tasks to be performed:
1) Connect your iPhone with your Apple CarPlay; (New User)
2) Open a music app (the one you usually use) and play your favorite playlist;
3) Adjust the volume to a comfortable level.
Scenario 2
The address of Joe’s place is 4300 Hartwick Road. Since you have never visited this neighborhood before, you need to find directions to Joe’s place.
Tasks to be performed:
1) Set your friend’s home address as the destination and start navigating;
Scenario 3
You are driving on the way to Joe’s place. Since the party is still one hour to go, the time is enough for you to keep the speed no greater than 35 miles per hour. When driving, you need to 1) keep at least one hand on the steering wheel all the time, 2) try not to look away from the road longer than three seconds and 3) never use your phone during the drive.
Tasks to be performed:
1) Skip the song and play another particular song (we will learn what songs are available on their mobile phone in advance);
2) You received a phone call from your friend and you have to answer it.
Scenario 4
Joe informs you through a text message that the party is now being hosted at a new address, 8150 Baltimore Ave. Now you are at a crossroad and its red light, so try to complete the following tasks within 1 minute before the light changes (But you can still continue if you need more time).
Tasks to be performed:
1) Check the message;
2) Send a message back saying that you will be there in 20 minutes;
3) Reset the destination and start navigating to the pub;
Scenario 5
Now you are on your way again. But suddenly, you realize that you’ve forgotten to take the housewarming gift with you, so you need to call your roommate to bring the gift to the address.
Tasks to be performed:
1) Call your roommate and tell her to bring the package to the address
We simulated the driving experience by asking the user to place one hand on the steering wheel at all times and look ahead as much as possible while performing a task.
Data Analysis and Overall Findings
The metrics used to analyze the test included both qualitative and quantitative measures.
After collecting the data we analyzed it using the following techniques:
Task Time & Completion Analysis
Time taken for each task and success/failure of the task performed was collected during the test sessions.
The results indicated Resetting Destination as the most time-consuming task leading to most task failures. Connecting with the Carplay, Setting up the navigation, Checking messages and Changing Songs were identified as other tasks of concern.
Task Difficulty Levels
After each task, we asked the participant to self-measure the ease of task on a 5-point Likert scale ranging from very easy to very difficult.
Connecting with the Carplay, Checking and replying messages as well as Resetting the destination were considered to be challenging. At least 40% of the participants found these tasks difficult or very difficult to complete.
SUS Score
After all the tasks were performed, we asked the users to fill out 10 system usability related questions based on the SUS Questionnaire. We calculated the SUS scores for each test participant and differentiated these scores based on users’ past experience with Carplay.
On average, new users gave a relatively low score of 65 to the Carplay system when compared to their experienced counterparts. It indicates a steep learning curve for new users and calls for a better onboarding process.
Affinity Analysis
We formalized the raw notes obtained from the usability tests, rewatched the sessions and created an affinity diagram to discover common insights and high-level findings.
Usability Issues
In order to better categorize the issues we discovered, they were sorted into two categories:
- Issues of high severity
- Issues of low severity
Issues of High Severity
- Connecting phones to Apple Carplay is difficult for a new user
Apple Carplay requires users to connect their phone via USB and there is guiding text on the landing page prompting users to connect via USB once they turn on Apple Carplay.
Some of the users failed to connect their phones to Apple Carplay. They did not notice the text on the screen, and almost instantly started to try and connect via Bluetooth, spending a lot of time exploring different pages on the system and buttons on the steering wheel trying to locate the ‘Bluetooth’ button.
Recommendation
Use animations to make the ‘Connect via USB’ guidance more noticeable
2. Checking messages can be hard and distracting
40% of the participants found it very difficult to check messages while driving mainly because the texts are not displayed on the interface but read aloud by Siri. This is different from most of the users’ conceptual model, who expect to see the text on the interface.
In addition, there is no way for users to go back to already read messages, which is also the reason why one user failed the task: the user didn’t pay attention to what Siri said when Siri was reading the message.
Due to these constraints, participants have to concentrate when Siri is reading the message to make sure they would not miss anything. Participants found this to be quite distracting.
Recommendations
Having both Siri reading the message and an option for users to see the content on the interface
Allow users to view/hear the already read messages
3. Resetting destination while driving can be inconvenient
Resetting the destination was found to be the most time-consuming task for participants. Only one user was able to finish this task within 10 seconds because she used Apple Maps — which supports Siri.
However, for users who are used to navigation systems like Google Maps and Waze could not use voice input because the apps are not integrated with Siri. This led them to input the address manually, which cost them considerable time.
Also, as users cannot read already read messages they could not go back and refer to the address they received as a message, leading to 2 participants failing the task as they forgot they exact address.
Recommendations
Incorporate Siri with other map apps
Let Siri automatically detect if a message contains an address and set the destination for users
4. Unintuitive interface design
Carplay’s interface provides a menu on the left showing a list of recently used apps. The one on the top is the app that is currently used by the user. It is an unclickable icon. The other two apps below the icon are recently used apps. However, during the tests, most of the users didn’t understand this design. They could not tell the difference between the icon and clickable buttons. Some participants tried to tap on the icon however nothing happened, which confused them.
The home button is located on the bottom left-hand corner. Two users had a hard time figuring out how to go back to the home screen because they did not notice the home button.
Recommendations
Clearly differentiate between clickable buttons and icons
Make the home button more prominent (such as in the center like many apple devices)
Issues of Low Severity
- Users want to get more control over Carplay notifications
Interacting with Carplay notifications led to a few unexpected usability issues.
One of the participants tried to swipe the notification to remove it but wasn’t able to do so as the notification system does not allow swiping off notifications.
2 of the participants were confused as the notifications disappeared automatically after a few seconds and they had to navigate to the desired app to see the notification again. This led to an increase in steps and time taken to complete a task.
Recommendation
Control over removing and viewing notifications
2. Need to make Siri more friendly and configurable for new users
We encountered a lot of issues surrounding Siri and how it should be made more friendly for new users.
2 of the 3 new users didn’t know with which apps Siri is compatible with.
In addition, all of the new participants didn’t know until the very end of the test on how to initiate voice control. Participants also talked about Siri not understanding foreign names and foreign accents.
Recommendations
Integrate Siri with other apps
More prominent Voice Control button on the Steering Wheel/Interface
Letting the users know if the app is Siri compatible or not
3. It takes time to find a particular song using Carplay
3 of the 5 participants took more than 40 seconds to find a particular song using Carplay. This was because most of the users used the Carplay interface to scroll through the list of songs — instead of using Siri — which was particularly time-consuming. This was again caused because participants were either using apps that were incompatible with Siri (such as Spotify) or just didn’t know how to initiate Voice Control.
4. Hardware can also be a barrier for users to complete tasks
Although this problem is not specific to Apple Carplay interface, it led to a lot of confusion among some of our participants. 2 of the 5 participants took more than 30 seconds to adjust the volume. This was because the users weren’t able to tell the difference between the two similar-looking adjusters as seen in the image above. It wasn’t clear which one of the two is a volume adjuster and which one is not. This was a clear problem with Volkswagen Jetta’s hardware.
Recommendation
Provide guidelines about how to better design hardware to incorporate Carplay
Limitations and Further Possibilities
- The driving experience was simulated and we did not make the users drive the car due to safety concerns. In the future, we can make the participant actually drive the car while using Carplay — under controlled conditions — and get even more accurate and relevant insights.
- Our tasks considered typical/ideal driving situations. Experiences may vary because of Road condition, Traffic, Weather, and Urgency. This problem can also be tackled in the future by making the participant drive the car on real roads without simulating the driving experience.
- Our study only targeted young, college-educated populations who are adept at using technology and newer interfaces (20–30-year-olds). Future studies could focus on a much more diverse population such as aging adults and/or disabled people. This would help us make Carplay more accessible to a wider range of populations.
