TransLoc Usability Test | Case Study
For my capstone project, my team and I did a complete revision of my previous usability test I conducted on the bus tracker app used by many college students, TransLoc. The original study contained many flaws that inhibited us from gathering accurate results, instead we refined our methodologies to make the study as comprehensive as possible. In this case study, I detail the ups and downs of the original and revised studies, the process, what contributions I made, and how I improved in the span of a year.
Members: 3-4
Roles: Notelogger, Observer, Writer
Methods: SUS, TAP, screening, pretest, & posttest questionnaires
Deliverables: IMRaD usability test report, project plan proposal, PowerPoint, individual memos
Duration: 2 weeks - 3 months
Overview
In the TransLoc Usability test, we were simulating being hired by TransLoc to conduct a usability test on the app and understand if there are any issues with it. TransLoc is a prominent bus riding app that is used by many students like myself yet makes it difficult to track buses, moreover causing students to be late to classes or miss their buses altogether.
The goal was to conduct a usability test on the app to identify users’ pain points and if their schemas match what the app has to offer.
Audience
​The intended audience for the usability test was to understand Kennesaw State University (KSU)’s commuter students’ – more specifically bus riders’ – opinions on the TransLoc app and their usage with it.
Role
My primary roles were the notelogger and observer. I took recordings of the participants during usability testing and took notes of their actions and emotions, both before and after testing then evaluated them. My other main responsibilities included;
​
-
Assisting the moderator and acting as a backup in case any issues arose.
-
Writing the project proposal plan addressed to our professor.
-
Co-writing the final report and calculating the final test scores.
Constraints
Throughout the duration of the project, there were constraints that kept the project from being as in-depth and accurate such as;
-
Lack of time
Plainly, 2 weeks to craft a fully actionable and well-executed plan was impossible. With 4 teammates putting in equal amounts of work probably would’ve helped bear the load, however that was not the case.
I was excited to do the usability test yet having to carry so many responsibilities between 2 people when it should’ve been 4 would’ve made the process smoother.
-
Unsupportive group members
There were times that one group member, Shania, wanted to leave because of the pressure, so having to manage her was an added load on my part. I understood her frustration but she was the only reliable member I had in the group. In the end, we did the best we could with what we had. It was a learning experience about usability testing and the work that goes into it, but it wasn’t an ideal situation.
-
Confusion / Not enough preparation
Another constraint being confusion was apparent once we saw the feedback our professor gave us on our IMRaD report that some information was miscategorized and should’ve been placed in “Methods” or “Results”.
It was very clear how unorganized the project planning was as we wrote the report in very little time which goes back to our lack of time.
Process
Preparation
Planning and questionnaires
1. We started off with planning the documents we would need before conducting the usability test such as drafting project plans, consent forms (for both researchers and the participant), SUS forms, and more. Regarding incentives, we decided making homemade cookies would be enticing yet cost effective for our project.
To continue, we prepared a screener questionnaire and moderator script to ensure we were being as ethical as possible while speaking with the participant.
It was a bit of a later addition, but Shania decided to include a pretest questionnaire to assess the participant on their mood and attitude before the testing. We already had a questionnaire after testing, so in hindsight, it may have been overwhelming for the user. Though it did give us some insight on the participant’s experience before and after testing.
Testing location
2. In relation to the study and our goals, it was important to plan where we would conduct our test. At first we thought going up to waiting bus riders at the main bus stop by the front of the school would be a good idea, but that would’ve been difficult to do as I was the only bus rider in the group.
We wanted to test the accuracy of the bus routes and have data from different phones, but again, I was the only bus rider, so it made that idea inactionable.
We finally settled on using the writing center on the Marietta campus to conduct our test since the study rooms were private and quiet. However, students don’t commute to the Marietta campus as much compared to the Kennesaw campus, so it made recruiting participants just as difficult.
User tasks
3. To continue, we worked on tasks that most bus riders would be looking to complete on the TransLoc app to reach their goal. In hindsight, it would’ve made more sense to make a few user personas and stories to narrow down what we’re looking for, but the tasks seemed manageable. I tried to use my own experience when helping to craft some of the tasks.
Instead of the tasks being listed for the participant to see, the tasks were given verbally from Shania, the moderator. To summarize, they involved;
​
-
Choosing a route
-
Tracking a bus route
-
Finding out when the next bus will arrive at a given spot.
Moreover, the entire moderator script allowed us to keep the testing schedule on track for each participant and tasks to understand their experience with the app’s functionality.
Role during testing
4. During this time, we decided who would take what role and what days we would be available for testing. I settled on being the notelogger and observer since I wasn’t comfortable talking to participants and a “people person” like the others were. Moreover, I think the role was fitting since I’m observant and good at taking reliable notes.
Screening process
5. The recruitment process for the participants was undesirable as we didn’t have a lot of time to develop a formal and thorough screening process. Shania recruited all of the participants from a D&D club. They were very helpful and really nice about the short notice for the testing; however it didn’t help whatsoever with what type of information we were looking for.
Again, having a user persona would’ve helped this as the participants weren’t necessarily bus riders, which was fine, but they were commuters who drove solo or by carpool. In retrospect, they didn’t qualify for TransLoc’s target audience and our usability test target audience.
Testing
Testing set up
6. On the day of testing, things would’ve been smoother with at least 3 helping hands such as being the moderator, notelogger, and observer but only Shania and I showed. This day was exceptionally important as it was getting close to the deadline and we didn’t have much for our report and needed our testing data as soon as possible.
That day, instead of being just the notelogger and observer, I also had to record at the same time too. Furthermore, we tested all 3 participants individually on the same day.
Additionally, the other 2 team members conducted testing on 3-4 participants on their own without our knowledge. There were no notes or script covering their testing so I chose to omit those participants and results but they are still mentioned in the report.
Usability testing
7. Moreover, we conducted the usability test in the following order for each participant:
1. The participant is given a consent form and recording consent form and asked verbally for consent.
2. The participant is given a pretest questionnaire on their mood before testing.
3. The participant is given their user tasks verbally while they complete the TAP process.
4. Afterwards, the participant completes their SUS form.
5. Participant is given a posttest questionnaire to further understand their insights on the app.
6. Participant is rewarded with homemade cookies.
Another point we didn’t think about was the Hawthorne effect which essentially states that one’s actions change based on being watched. The room was rather small, so I could only record from the back of the room. I believe this also tainted the test results just as much.
Presentation
8. After we collected our data, we worked on the report and presentation. The other teammates helped a bit but still, a majority of the work fell on us. Finally, we showed off our findings in a final report and a presentation but could’ve been better and structured effectively.
Retrospective
In summary, my team and I conducted a usability test on the TransLoc app, an app used to monitor bus routes from nearby schools, like KSU. The main audience was to focus solely on KSU students who’ve ridden the Big Owl Bus (BOB) before or at least ride it daily. Though the preparation and testing seemed doable, our lack of time and lack of support from 2 of the group members made the project rather difficult.
Through some of the issues, Shania and I worked on preparing the testing documents, writing the report, conducting the usability test and more. There were a few times where the pressure got to Shania, but I reassured her that we would be able to get the project done in time, which we did.
The project was not very ideal but it gave me a glimpse to what issues can occur with usability testing and how each member needs to pitch in to make the project as efficient as possible. A good takeaway for me was understanding how to cope when an issue arises and what I can do to improvise.
However, in the span of a year, I decided to do a complete revision of the project and made drastic improvements from my past mistakes and incoherent results as the project lead for my senior capstone project.
TransLoc Usability Test | Revision
Members: 3
Roles: Project Lead, UX Designer, Moderator, Notelogger, Observer, Editor
Methods: SUS, TAP, Likert scale, closed card sort, survey, heuristic evaluation, simple random sample
Tools: Figma, Microsoft Word, Microsoft Forms, wireframes, mockups
Deliverables: IMRaD usability test report, proposal memo, PowerPoint, mockup slides
Duration: 3 months
Updates
Overview
For my senior capstone project, I decided to revise my TransLoc project to make it more comprehensive than what it was before. There were some elements that went right, however, the amount of time the group was given and available resources and skills detracted from how well the project could’ve been.
Instead, my group and I decided to redo the usability test and added more methods to better align ourselves on what specific information we were looking for. With the added time of 3 months, we were able to present a better usability test on the TransLoc bus rider app and report.
Audience
Instead of the audience being solely focused on commuter students, it made more sense to open the scope to focus on KSU students being commuters and bus riders, or other methods of transporting to school rather than specifically bus riders. Also considering that there are buses to take students to further parts of campus such as West Campus and Big Shanty Loop where you can commute to campus by driving and still ride the bus.
There are many nuances and opening up the scope of the audience seemed more fitting and less complicated when it came to recruiting participants for the test. Also understanding who the audience is was just as important judging from the diverse usages that come from the TransLoc app.
Role
This time I was the project lead since I had knowledge of what elements could be changed from the previous TransLoc project. In addition to team lead, I was the notelogger, observer, and moderator since we had ample opportunity to take turns during each session.
To add, I had to take on more responsibilities such as;
-
Creating the redesign mockups and wireframes for the app through Figma.
-
Managing my groupmates on our progress.
-
Designing a planner and coordinating our schedule to keep us on track.
-
Presenting our ideas effectively during the showcase.
Despite my initial worries, the role was easy yet frustrating at times because of the amount of work I had to do.
Methodology
We kept our previous methods from last time such as TAP and SUS yet we added more methodologies to the mix such as conducting a heuristic evaluation, and including a Likert scale, closed card sort, and survey for added metrics on demographics of the KSU students’ usage of TransLoc and how frequently they ride the buses, if at all.
Last time, we had a hard time recruiting people with only 3 participants, however this time we used a few sources to send out a survey to students around campus. Utilizing some of my research in Tech Comm notes, we were able to help craft a useful and straightforward survey for people to take in only a few minutes. In turn, we got 30 responses by sending out links through our GroupMe chats and the school’s subreddit on Reddit. So, the pool of data we collected was far more diverse than it was before.
Additionally, to implement these ideas for changes, I created static mockup revisions of the homepage and search page of TransLoc given the feedback we received during the usability tests and our heuristic evaluation.
Process
Preparation
Brainstorming
1. The start was a little slow but we quickly picked up speed by brainstorming ideas of what our final project could be. After giving my feedback on what past projects we could do, we all settled on redoing the TransLoc usability test.
We had a bit of trouble planning out how we wanted to start, what changes needed to be made, and so on. We relied on our notes and guides from previous classes and external resources like Usability.gov to help us get started and develop our proposal plan memo.
​
In an effort to make the revision of the project more organized and less overwhelming, I developed a cohesive planner to keep track of upcoming due dates and allow us to mark off completed tasks easily. It wasn’t a major part of the project, but it allowed us to stay on track and accountable for what responsibilities we had.
User personas
2. To help us get our bearings about who the project should focus more on, we created user personas with user stories based on the target audience for TransLoc and the usability study. For the previous project, this is something I wish we would’ve done as it helped guide us in who we’re looking to test on.
Here is my persona, an international student who’s finishing up their degree at KSU (loosely based on one of my friends). I designed him an international student to better represent the student body who use the Big Owl Bus (BOB) as there are many international students at KSU who also use the buses, a missing demographic from the previous project. Additionally, my teammates developed personas as well.
Also as a quick measure of accuracy, I went and asked our professor, who previously taught us in a usability class, to check our user personas. Before using the data in the report, I wanted to ensure that the personas were as detailed as possible to help readers understand how the personas guided us to the target audience, user tasks, and survey portion.
User tasks
3. Before starting testing, we made sure to create the consent forms, a survey, and testing documents we need. Additionally, we used that time to individually perform heuristic evaluations on the app to understand what the app does well and what it doesn’t. After a few days, we came together and compiled our findings into multiple lists then defined the issues based on severity levels ranging from cosmetic to catastrophic.
Judging from the issues we had, we used that information to create 4 sets of user tasks for each participant to complete during the usability test (3 out of the 4 tasks).
Survey
4. In this part, we developed the survey to send out, which contained about 8 questions to get relevant information such as demographics and usage of the TransLoc app and KSU’s BOB. We were able to receive 30 responses from a multitude of KSU students ranging from freshmen to seniors and from ages under 18 to 44 and older. I was really excited to see this as this meant our data was extremely diverse by featuring a wider range of the school’s representation than what was shown before in my previous version of the project.
Due to financial restrictions for incentives, we could only select 3 participants for the test so we conducted a simple random sample to make the selection process as fair and error free as possible.
Constraints
5.
-
Lack of emails received from potential participants.
Together, a teammate and I emailed 6 qualifying participants and only got 2 responses. I was surprised but content that we got a lot of responses in case a participant failed to respond.* This caused a slight delay in the testing schedule and pushed us back from starting on the report sooner.
*Only those who used their KSU student emails opposed to using their Gmails as this would affect the ethics and integrity of the communications.
-
Unexpected addition of a tester.
That being said, we had 2 interested participants and decided waiting until the following week to start testing would be easier, yet at the last minute a teammate asked a classmate to be a tester. I didn’t agree with this idea since it felt like it would skew the results somewhat. The classmate was familiar with the app and a former bus rider which was helpful, but still a last minute idea. It was only after testing that the unexpected participant took the screener questionnaire unlike the other participants.
Prototype wireframes and mockups
6. During the week the survey was being sent out and completed, I started on the wireframes and mockups for revisions that could be made for the app. Getting a head start was essential before testing, so when we hear user feedback on the app I can make additional edits in a quick manner as we were getting close to the deadline. Besides, I was the only one in the group who knew how to use Figma so the designing part fell on me and was essential for the presentation visuals. Overall, these were the end results of the revisions.
Wireframes
Mockups (iPhone left, Android right)
Additionally, after testing I went back and added elements participants mentioned being an issue during usability testing for example, one participant stated that it would be helpful if selected bus routes on the map had labels.
Testing
Testing set up
7. After making room reservation arrangements and coordinating testing schedules with participants, we began testing. This time was a bit different than the first as the room was bigger, private, and allowed more researchers to be in it at once. Though, the Hawthorne effect was still an issue like the previous project.
We decided to test each of the participants individually which was helpful mainly to prevent any issues of the participants changing their answers depending on the others’ opinions and so on. Besides, it gave us time to reset the room before the next test.
Usability testing
8. For each participant we allowed them to use our iPhones to use the TransLoc app. The reason for this was to remove any possible complications such as previous trip data and remove inconveniences for the participant. This was different from what I did last time with my group where the participants used the app on their own phones. Though, it would’ve been nice to have some diversity in device types for the usability test.
To continue, after the participant used the app while attempting the 3 out of 4 user tasks and using TAP during the process, they were given the Likert scale, SUS, and closed card sort sequentially.
Roles during usability testing
9. During the testing process, the researchers and I were able to switch between roles of notelogger, moderator, and observer. The experience was nice to have and being a moderator was new to me. We didn’t have a script, but made sure to ask the participant about their decisions, what they thought could be improved for the app, and more. Overall, we remained unbiased in our word choices and allowed the participants to think for themselves without too much guidance.
Report after testing
10. After the testing finished, we gave the documents to one of our teammates to start dissecting the data, and also start on the report.
We initially divided up the work while working on different sections of the IMRaD and I had the discussion section explaining the results from the data and how it led to us coming up with solutions for improvements with the app. After the writing was done, I went in and edited the 30 page report (yes, that long) for consistency and including information to give detail about the preparation, testing, and evaluation processes. Admittedly, it was tough trying to structure the report according to IMRaD and APA guidelines, but I tried my best.
*Documentation for tests can be found in the Appendix of the TransLoc report.
Presentation
11. We presented our findings during our senior capstone showcase and got feedback on our work, for example, improving the accessibility of the mockups as some of the colors from the color palette conflicted with the white text.
This gave me some food for thought as I want to go back and make edits then write about it through a process page like one of the guests told me.
12. My teammates prepared the deliverables such as a PowerPoint for the report showing the highlights and printed copies of our executive summary for our report while I made static mockup examples of the app revisions in short Canva slides.
Retrospective
I’m really content with my decision to redo my previous TransLoc project for a new, upgraded version. I was interested in the project initially given it’s the only app for KSU students to track the Big Owl Bus (BOB) yet works so poorly. To add, we wanted to understand user’s pain points with the app and understand ways that it can improve for a better user experience.
Compared to the previous project, the revision felt completed and explanative of the methodologies we used and how we came to the conclusion of our results. Additionally, the revision allowed me to improve myself by becoming the project lead and adding onto improvements such as creating static mockups of the homepage and search page of TransLoc.
Overall, it would’ve been nice to use the mockups for a potential A/B test, however I did have a concern about overwhelming the user with too many tests at once as they already had to do 4. On a positive note, instead I thought it would be worth it to take the user feedback and adjust the mockups to fix issues the users had while using the app.
To conclude my experience, there were many benefits that came from the capstone project revision for example:
​
-
Developing my leadership and presentation skills
-
Improving my writing/editing skills
-
Explaining my arguments for usability improvements
-
Improving my prototyping skills
Moreover, I’m glad I used the opportunity for my capstone project to show my growth on how I improved my skills and corrected mistakes from my previous usability test.
![IMG_0315[1].jpg](https://static.wixstatic.com/media/c36dd0_1f28601aae1e4d188cee45bf5314c5d2~mv2.jpg/v1/fill/w_901,h_676,al_c,q_85,usm_0.66_1.00_0.01,enc_avif,quality_auto/c36dd0_1f28601aae1e4d188cee45bf5314c5d2~mv2.jpg)