Caltrain Mobile App

Timeline

August - December 2022

Role

Team Lead, UX Research, UX Design

Team Member

Ariel Tam

Deliverables

Report, Presentation, Poster

Tools

AppFollow, G Suite, Figma

The Caltrain Mobile App is designed to provide users with easy access to mobile ticketing services for the widely used transportation option in the San Francisco Bay Area.

Context

Caltrain's mobile app, designed primarily for ticket purchases, often falls short of user expectations. Many users desire a more integrated trip-planning tool, and frequent app crashes, especially after buying tickets, are a major concern.

My Impact

As the team lead, I successfully managed a project with a reduced team due to the absence of one member. I developed and implemented a comprehensive project timeline, ensuring efficient task allocation and timely completion.

Overview

Caltrain, a widely used transportation option in the San Francisco Bay Area, offers a convenient way for commuters to travel within the region. The Caltrain Mobile App is designed to provide users with easy access to mobile ticketing services, allowing them to purchase and store their tickets on their mobile devices, making their transit experience more seamless.

Problem

Users find that Caltrain's mobile application does not provide satisfactory features for trip planning.

Process

Discovery Research, Prototyping, Usability Testing

Solution

Ticketing

The purchasing process now requires only four steps on a single page and easily navigate between steps without relying on the back button.

Organized stations

Stations are organized sequentially by zone.

Accessibility information for each station is indicated on the right side.

Trip planning

Users can find stations based on their current location or a specified address and departure or arrival times. A new "Train Status" tab provides real-time updates, live tracking, and scheduled train times.

Outcome

The redesign led to a 54% and 74% decrease in average completion time for tasks involving trip planning which indicates that trip-planning tools in the app became more supportive and effective for users.

This study was conducted as a project for the Human-Computer Interaction class (ISE 217) at San Jose State University.

View the Full Report

View the Poster

The Problem

How do we improve the "on-the-go" experience for users using the app?

The main question can be broken down into three guiding questions:

What makes users use Caltrain to get tickets vs in-person ticketing kiosks?
What makes users want to view train schedules and stations from the app instead of the Caltrain website?
What makes users want to continue to use the Caltrain app?

Our review analysis uncovered significant user dissatisfaction with the Caltrain Mobile App's limitations. While ticket purchasing works well, the app falls short in trip planning. Displaying train schedules and routes through an in-app browser that redirects to the Caltrain website is a major pain point. Many users questioned the app's value beyond ticket buying.

Our usability testing corroborated the user reviews. While the ticket purchasing process had minor issues, the app's core functionality in this area was acceptable. However, users experienced significant frustration when attempting to plan their trips.

Discovery Research

Understanding the problem.

App Store Reviews

The Caltrain Mobile App boasts a stellar 4.77-star rating on the app store, but a closer look reveals a different story.

Many 5-star reviews might not reflect the app's true usability as with the examples below.

To understand user pain points, my team and I dug deeper into the reviews, analyzing user sentiment beyond just star ratings.

Interface Evaluation

Our evaluation of the Caltrain Mobile App identified both strengths and weaknesses in interface design. The app effectively applies Fitt’s Law by using a large, prominent "Buy Products" button. The bottom navigation bar also adheres to best practices by featuring the core functions of "Products," "Trip Tools," and "Account."

However, the interface violates Gestalt principles in several ways. Inconsistent indentation in the station list creates ambiguity about station groupings, and the placement of the wheelchair accessibility icon further exacerbates this issue. Users may mistakenly perceive stations without the icon as headers for station groups.

Additionally, the Hick-Hyman Law suggests that the number of options can impact response time. The 31 stations displayed on the origin and destination selection screens may increase user decision-making time.

South San Francisco and Broadway can be perceived as the headers for a set of stations.

An extensive list of stations could cause confusion for novice or intermediate users who are unfamiliar with the station names or order.

Usability Study

Our usability test evaluated the Caltrain app's performance on five core tasks using time to completion, error rate, and user confidence as metrics.

Tasks used in the usability testing

Buy a train ticket for an upcoming trip. Your closest station is Sunnyvale and your destination is San Mateo.
Buy a parking permit for the San Bruno Caltrain lot. Your space number is 028.
View the train ticket you have just bought.
Find the closest Caltrain station to your location and destination. Your location is at the College of San Mateo and your destination is the SAP Center in San Jose.
How do you get to San Francisco station by 11:30 AM on a Saturday from Mountain View? What time is your train and how long will your trip be?

While users successfully completed ticket purchasing tasks (1-3) with reasonable efficiency and confidence, significant challenges arose when planning trips (tasks 4 and 5).

Findings from the discovery research

App Reviews

The Caltrain mobile app, currently limited to ticket purchases, falls short of user expectations for a comprehensive trip planning tool. While the app offers basic trip planning functionality through an in-app browser, many users have explicitly requested a more integrated solution.

Despite its primary function, the app's reliability is a significant concern. Numerous reviews highlight frequent crashes, particularly problematic when users have already purchased tickets that become inaccessible. For a single-purpose application, this level of unreliability is unacceptable.

Our analysis revealed a recurring theme: slow performance and app crashes, which significantly impacted user satisfaction. The increasing number of negative reviews suggests that the current features of the Caltrain app are falling short of user expectations.

Interface Evaluation

Strengths

Effective application of Fitt's Law: The "Buy Products" button's prominence aligns with user expectations and facilitates easy interaction.
Adherence to best practices: The bottom navigation bar's clear organization of core functions enhances usability.

Weaknesses

Violation of Gestalt principles: Inconsistent indentation and icon placement create confusion and hinder user understanding of station groupings.
Potential for increased decision-making time: The large number of stations displayed on selection screens may slow down user interactions.

Usability Study

Users struggled to find relevant information and complete trip planning tasks within a reasonable timeframe. Error rates were notably higher for these tasks, and users expressed frustration and uncertainty about the process. For instance, one participant commented, "I couldn't figure out how to find a train that would get me to SF before 11:30 AM."

These findings indicate a critical gap in the app's functionality. The absence of essential features like real-time transit information, alternative routes, and clear journey planning options hinders user experience and may discourage continued app usage.

Who are our users?

To deeply understand Caltrain riders, we created personas representing novice, casual, and knowledgeable users. We developed empathetic representations of our ridership by analyzing App Store reviews and the Caltrain Customer Satisfaction Survey.

Expert User

Kira Francis, 30

Software Engineer in Menlo Park.

Lives in San Mateo.

Kira moved to the Bay Area and has been in the area for four years. She rides the Caltrain almost daily to work. She likes using her morning commute to review her emails to get her ready for the day.

Knowledgable User

Don Nguyen, 62

Retiree.

Lives in San Jose.

Don is a Bay Area native. He is a retired teacher and he lives with his wife. His children have all moved out of state but they visit a number of times a year. When they visit, Don likes taking his grandchildren to the Giants’ games in San Francisco.

Novice User

Andrea Park, 19

Student at Mission College in Santa Clara

Lives in Santa Clara

Andrea is a new student at Mission College. She had just moved to the Bay Area from Portland. She loves photography. In her spare time, she plans to explore San Francisco to photograph the scenic views.

Insights from our discovery research

By developing personas, we gained insights into the needs of Caltrain users. The existing application demonstrated potential for aiding trip planning, so we focused on enhancing this feature. We identified opportunities to improve the ticket purchasing flow by aligning it more closely with real-world processes and preserving ticket purchase information for users.

Suggested usability enhancements:

Buy Tickets:

Reorganize the station lists and allow users to search train stations.
Improve visual contrast for descriptive content.
Include clearer instructions on the terms of usage of the tickets.
Option to receive an email confirmation with the ticket's QR code.

Trip Planning:

Incorporating trip planning tools such as maps and route information.
Train live-tracking and status notifications.

Prototyping

Implementing changes based on the suggested usability enhancements.

The streamlined purchasing process requires only four steps on a single page. Users can easily navigate between steps without relying on the back button.

Stations are organized sequentially by zone. Accessibility information for each station is clearly indicated on the right side for better readability.

This iteration offers enhanced trip planning capabilities, allowing users to find nearby stations based on their current location or a specified address. Users can also plan trips based on departure or arrival times, a feature particularly useful for novice and intermediate users. To provide more comprehensive travel information, a new "Train Status" tab has been added to the navigation bar, offering real-time updates, live tracking, and scheduled train times.

Usability Testing

The study results showed that users completed tasks more quickly overall, except for the second task. This could be attributed to the prototype's limitations and the participants' expectations during the usability testing. Additionally, tasks 4 and 5 were identified as challenging during the preliminary usability testing, but the redesign improved the average completion time by 54% and 74% respectively, as well as decreasing the error rate and increasing users' confidence levels.

These findings indicate that the trip-planning tools in the app were more supportive and effective in helping users plan their trips.

Final thoughts

App performance is critical for user satisfaction. Reliable app functionality, including speed and stability, directly impacts user experience and retention. Issues such as crashes or slow load times, especially when users need to access essential information like tickets, can lead to significant frustration.
Users expect comprehensive functionality. Beyond core features like ticket purchasing, users seek additional value from transportation apps, such as trip planning tools. These features enhance the overall user experience and encourage continued app usage.
Tailored experiences are essential for diverse users. Different user groups have distinct needs and preferences. Providing shortcuts for frequent users and clear instructions for new users improves app usability and satisfaction.

Project limitations

High-fidelity prototype limitations restricted user interactions and context.
Incomplete train schedule functionality hindered testing and caused confusion.
A small sample size limits the generalizability of findings.
A controlled testing environment may not reflect real-world usage.

What I would do if I had more time...

Expand platform coverage to the Android OS to ensure design consistency and usability across platforms.
Conduct field studies to observe user behavior in real-world settings to better identify pain points and opportunities for improvement.
Implement surveys to gather quantitative data on user satisfaction, preferences, and overall experience.
Increase sample size and expand the participant pool for more robust and representative findings.
Continuously test and iterate on the design solution.

Page updated

Google Sites

Report abuse