Mattermost Pilot Evaluation

The Mattermost pilot began its formal evaluation phase in September 2017, and went through the 2017W1 and 2017W2 terms. 10 courses from Faculty of Education and Arts participated in the evaluation. Seven instructors and one TA gave in-person feedback, and over 200 students completed online surveys about their experience.

Letitia Englund, a UX/UI Analyst from UBC’s Centre for Teaching, Learning and Technology led the evaluation phase of the project, and prepared an official evaluation report summing up the findings. The report is available upon request from the LT Hub. Joe Zerdin and Ian Linkletter also contributed to the report.

The next four sections of this page are quoted from the Executive Summary of the evaluation report.

Instructor Experience

From the evaluation report:

“Instructors generally noted pedagogical value in Mattermost for teaching because it helped keep communication flowing in their courses and cohorts. Most instructors felt Mattermost specifically improved 1-on-1 communication between themselves or the TAs and the students (particularly through private channels and direct messages), and many appreciated the opportunity to crowdsource basic answers to course questions by inviting students to interact with one another in public channels. Some instructors also thought Mattermost encouraged more informal conversation (both academic and non-academic) that fostered a sense of community, presented a modern interface with interactivity relevant to today’s workplaces, and had the potential to offer learning spaces that could persist for students outside an individual course context.”

Student Experience

From the evaluation report:

“Some students thought Mattermost offered: 1) a stronger sense of connection with other students (due in part to improved ease of answering course-related questions with the help of peers), 2) several usable and useful features (in particular, timely notifications to draw the student’s attention back to the course), 3) an immediacy to interactions with other students and instructional teams that helped remove work blockages more quickly, and 4) in the most active course specifically, a way to engage more informally around a course.”

Instructor Challenges

From the evaluation report:

“Instructors struggled most with the real-time chat aspect pushing them to be more fluid with their time and work/life boundaries. Some were fine committing more time and energy to monitoring and responding to Mattermost; others pushed back (and this may account for some of the lack of uptake in those courses). Instructors also found it challenging to know how much to drive students to use the tool, recognizing that students are already weighed down by multiple technology platforms and outside commitments. And, as with instructors themselves, they wanted to respect not all students naturally gravitate toward this medium as a means of communication. Finally, some instructors wanted more functionality (especially better participation and activity statistics) from Mattermost to be able to use it more robustly as a teaching and research tool.

Student Challenges

From the evaluation report:

“Some students perceived that Mattermost presented: 1) an unwelcome additional external platform to access and log in to (with added frequency given its real-time nature), 2) a heightened sense of communication overload by adding this to the plethora of existing personal, professional, and academic communication mediums, 3) a few features with poor usability (or initial learnability) and in particular, channels becoming harder to parse the more content gets added, and 4) a feeling of being left out of discussions when timing seemed off for participating (whether due to time zone differences, non-academic commitments, or a desire to reflect more deeply first).

Recommendations

The evaluation report generated recommendations for “how Mattermost could best be implemented at UBC to maximize its perceived benefits and minimize its perceived shortcomings”.

  1. Set up for smaller groups of students at once »
  2. Organize content into clear channels and guide shared organization »
  3. Set expectations around instructional team availability »
  4. Integrate or regularly prompt to promote use »
  5. Set loose guidelines for student participation »
  6. Emphasize private communication options, especially for fully online »
  7. Explain why Mattermost over other more established chat tools »

Learn how to implement these recommendations in your course on our Mattermost Recommendations page.