An Evaluation Report
From: Group 1 TWC 301
To: Dr. Shelley Rodrigo, CIO
GEEK TECH UNIVERSITY IT DEPT
TABLE OF CONTENTS
EXECUTIVE SUMMARY 3
INTRODUCTION 4
DEFINING THE PROBLEM 5
CRITERIA 7
DISCUSSION OF OPTIONS 8
COMPARISON 9
CONCLUSIONS 10
FINAL RECOMMENDATIONS 11
REFERENCES 12
APPENDIX 13
EXECUTIVE SUMMARY
Due to the continued expansion and growth of Geek Tech University, the GTU IT Department submitted a proposal requesting permission to review different project management tools (PMTs). The PMT would enable the department to streamline IT communications, allow data transfer between departments and facilitate file transfer. Current flow of information in the existing system is hindering the daily operations of several departments as well as creating unnecessary expenditures.
The IT Department received a positive response to the proposal request and evaluated three PMTs to determine the best application to fulfill GTU needs. Four GTU technicians were asked to evaluate specific functions within each of the PMTs. PMTs were evaluated for ease of use, management features such as file transfer capabilities and implementation costs.
After conducting a usability study of three PMTs, the GTU IT Department has determined that Smartsheet provided the features most beneficial to its needs. Smartsheet allowed the technicians to communicate using the PMT and complete all of the tasks as assigned in the most time efficient manner.
It is the recommendation of the GTU IT Department that GTU secure the services of Smartsheet. By using a centralized and web-based communications and information system project deadlines were met or exceeded through specific task management, file sharing was simplified and interdepartmental communication and overall productivity was improved.
INTRODUCTION
The purpose of this evaluation report is to provide information regarding the usefulness of online project management applications in assisting with the completion of large-scale projects at the GTU IT Department
At the request of the GTU IT Department, Project Management Tools were evaluated in order to alleviate data management concerns including:
• Database security
• Loss of accountability
• Version-control issues of documents and files
• File disorganization
• E-mail communication
• Failed updates or missed file transfers
• Miscommunication within department personnel
• Meeting project deadlines
• Task Management
Three PMT web based sites were chosen for evaluation: Basecamp, Zoho and Smartsheet. The applications were chosen for their advertised potential of meeting the IT department needs. The functions the IT department specifically sought to utilize included:
• Code-encrypted file sharing
• Web-based mass storage and access
• Email notifications
• Deadline and meeting alerts
• A history-tracker for changes made to documents and projects
• User-friendly templates
• Calendar functions
A usability team of IT technicians was created to evaluate the chosen PMTs based on the criteria as listed above.
DEFINING THE PROBLEM
The GTU IT Department has grown threefold over the last 5 years. It is one of the largest university-based IT support teams in the US, employing over 300 employees nationwide. The financial impact of our exponential growth has been very positive, but has presented the IT Department with operational challenges. Under the antiquated project management system (GeekWorks 2.1) in place since 2008, we are unable to effectively manage and support large-scale projects within the department. Our entire staff, especially those in remote locations, have expressed valid concerns with communication and task management relative to the current system.
As a result, The GTU IT Department has suffered a decline in productivity due to access problems with task and project lists. Data that is not well-managed has resulted in compromised database security, loss of accountability, version-control issues with the documents, and disorganization of files. Further, GTU IT technicians share information, track projects, and communicate solely via e-mail. Miscommunication problems have developed extensively from lost e-mails, failed updates or missed file transfers. A number of large project deadlines have been missed. These missed deadlines have resulted in financial loss. The Data representing missed deadlines and the cost analysis is provided below in Figure 1.
In addition, GTU projects a major expansion within the next five years, beginning with the construction of a new downtown campus in September 2010. Accordingly, IT technicians and support teams will be expected to spend significant time working off-site. In order for the IT Department to maximize productivity and efficiency, a remote office location is planned. It is critical that we address our current system’s problems expediently, in order to have the solution in place on or before August 31, 2010.
The following chart (p. 6 fig. 1) illustrates the financial impact of missed project deadlines as a result of communication breakdown from the limitations of GeekWorks 2.1:
IT Department Missed Project Deadline Analysis
January 1, 2009 - January 31, 2010
Project Original Deadline Date Completed Technician and Support team OT hours Total Additional Cost
Google Calendar Synchronization 14-Mar-09 15-Mar-09 7 371
IT Division manager reports failure to receive voicemails regarding issue with Daylight Savings Time. IT Division manager was on site at Colorado regional retreat, and cell phone was out of range.
Microsoft Office 2007 Project Rollout 1-Apr-09 15-May-09 63 3276
Project Manager reports missed crucial updates not installed, Incorrect edition of Microsoft Exchange 2007.
YouTube Training Videos – Usability Test 1-Jun-10 13-Jun-10 12 636
Project Manager reports miscommunication between training department deadline and IT department deadlines. Training Department instructional material was not presented to IT department until June 7. Training Department did not receive IT Department email regarding new June 1 deadline.
Quickbooks 2009 Update 1-Sep-09 4-Sep-09 26 1378
Additional Accounting Department Support team hours 38 1330
Accounting Team reported request for change in Quickbooks update deadline date via email on August 31. IT Department was in process of in-field project and missed communication. Resulted in additional hours spent to install update. Additional Accounting Department hours required to correct errors due to missed update.
Figure 1.
CRITERIA
The online project management tools were compared by required functionality and characteristics and each product was scored by comparison.
• Cost of implementation (Purchase, training, installation, maintenance)
• Ease of use
• Differentiation from Geekworks 2.1
• Support Requirements
• Available Features
REVIEW OF OPTIONS
Project management tools provide the resources for planning, organizing, and managing data, information and people together to bring about the successful completion of specific project goals and objectives. However, each application reviewed has its own dynamic qualities and/or limitations. The foremost challenge of these applications is to achieve all of the project goals and objectives while working within project constraints including scope, time, and budget.
ZOHO
Zoho offers a comprehensive suite of award-winning on-line business, productivity & collaboration applications. Customers use Zoho Applications to run their business processes, manage their information and be more productive while at the office or on the go, without having to worry about expensive or outdated hardware or software. Primary benefit: clients can customize particular applications to their needs as opposed to paying for a comprehensive plan that includes items they may not otherwise use.
Pros: Data security, control of your data, integration, multilingual, and customer friendly support.
Cons: Individual application download time consuming, not comprehensive for budgetary constraint, and complicated.
BASECAMP
Basecamp is the leading web-based project collaboration tool that allows users to share files, meet deadlines, assign tasks, and centralize feedback. Primary benefit: Leaders in the market, and have developed a tested and secure system used by many different companies.
Pros: Project development tracking, enabled collaboration, email reminders, visible progress of projects.
Cons: Does not allow you to assign more than one person to a task, no calendar on the dashboard, and plans are expensive.
REVIEW OF OPTIONS (cont.)
SMARTSHEET
Smartsheet Software as a Service (SaaS) provider gives people a fast, easy way to collaborate and get things done. Smartsheet makes individuals who work with others more productive through an innovative, easy-to-use online application.
Pros: Comprehensive package at the most affordable rate that met within budgetary restraints, offers fast upload of files, data encryption, track discussions, allows users to share online with co-workers, clients, contractors, and vendors, and sends email reminders and notifications of updates/changes.
Cons: No deadline alerts.
COMPARISONS
Project Management Applications
Basecamp Smartsheet Zoho
Features
Task Management * * *
Calendar * * *
Discussion Board *
Email Notification *
Deadline Alerts * *
Meeting Alerts * *
History Tracking * *
Templates * *
Cost for license $600 $325 $1,200
Monthly Fee 0 $50 0
Trial Period 30 Days 14 Days 30 Days
CONCLUSIONS
The results of the usability test concluded that 4 out of 4 users were extremely satisfied with Smartsheet. Smartsheet allowed the Usability Team to effectively maintain communication during the project. Tasks were completed in a timely manner. Task management tracking enabled team members to view what portion of the project had been completed and by which team member. E-mail notifications were sent out to team members whenever changes and updates had been made. The Discussion Board allowed for team members to upload files and make comments and notations to fellow team members. File transfers were not limited in file size and assigned to specific sections in order to work on multiple projects if necessary. Calendar and Gantt functions allowed for deadlines to be maintained and e-mail notifications of upcoming deadlines were also provided through Smartsheet. In addition, Smartsheet costs were well within budget limitations as set forth by GTU.
RECOMMENDATIONS
Based on a thorough examination of the top three project management tools, we recommend Smartsheet. While Smartsheet is the only project management application that charges a monthly fee, it is also the only one that provides a discussion board and e-mail notifications. The results of the usability test confirmed that users believe that discussion boards are beneficial to collaboration.
Smartsheet combines spreadsheet-like ease of use with powerful work automation and file sharing – all integrated with data resources. Its online work management tool takes an easy to use spreadsheet to the next level and is compatible with QuickBooks data import, task management, alerts & reminders, calendar and Gantt views, file sharing and discussion tracking.
Overall Benefits
• Easy to use spreadsheet-like interface ensures team adoption and easy tracking of work
• Import of QuickBooks data (with optional filters), including customer and vendor lists, open invoices, estimates, bill and sales orders
• Track any type of work, including projects, task lists, sales pipelines, IT issue logs, HR checklists, marketing programs, and team goals
• Collaborate securely on shared work or task lists, attached files and discussion threads
• Increase accountability with a time-stamped history of all changes and automated alerts & reminders
• Ensure nothing falls through the cracks with summary reports across all types of work.
REFERENCES
• Dobrin, Sidney I., Christopher J. Keller, Christian R. Weisser. Technical Communication in the Twenty-First Century. Columbus, OH: Prentice Hall, 2007
• Dejean, David. “Online Office Apps Get Real: Google Docs vs. ThinkFree vs. Zoho.” Computerworld. 16 July 2008. 14 April 2010. http://www.computerworld.com/s/article/9108799/Online_office_apps_get_real_Google_Docs_vs._ThinkFree_vs._Zoho
• Wenzel, Elsa. “Alternatives to Microsoft Office.” CNET Reviews. 24 May 2006. 14 April 2010. http://reviews.cnet.com/1990-3524_7-5140428-1.html
• Basecamp. 14 April 2010. http://basecamphq.com/
• Smartsheet. 14 April 2010. http://www.smartsheet.com/
• Zoho. 14 April 2010. http://www.zoho.com/
• Reynolds, Deanna. “The Top Ten Benefits of Project Management.” Bright Hub. 28 July 2009. 3 May 2010.
http://www.brighthub.com/office/project-management/articles/2350.aspx
APPENDIX
USABILITY TEST REPORT
FOR GTU IT DEPARTMENT
PROJECT MANAGEMENT TOOL EVALUATION OF:
BASECAMP, ZOHO, SMARTSHEET
Created by Group 1
Valerie Cooper, Jennifer Nguyen, Kathryne Nunley, Osmara Altenhoff
Geek Tech University, IT Department
May 02, 2010
EXECUTIVE SUMMARY
This study was conducted to test the usability success of Basecamp, Zoho, and Smartsheet for possible implementation of these web-based project management applications at GTU IT Department, replacing the current project management system, Geekworks 2.1. The goal was to measure and identify user satisfaction, ease of task completion, problem areas or inconsistencies with these applications by asking test participants to use the most commonly used features or functions of these web-based applications in relation to projects that the GTU IT staff work on. Participants completed these tasks at their homes, with the use of their personal computers during their spare time. Four participants from Geek Tech University IT Department were recruited and screened, based on various computer skill levels and project responsibility.
All participants were required to complete tasks within three days and for the most part, found the web-based applications simple to use, intuitive and agile. Questionnaires were given to the participants before and after the test were conducted. Participants were asked to elaborate on their test experience via instant message or email. Solutions and recommendations leading to a decision as to whether or not these tools will replace Geekworks 2.1 will be determined.
INTRODUCTION
This report describes the plan, execution, and results of conducting the usability test for Basecamp, Zoho and Smartsheet web-based project management applications.
USABILITY TEAM
Test administrators and report writers included Geek Tech University’s IT Help Desk Manager, Valerie Cooper and IT Department partners Osmara Altenhof, Kathryne Nunley, and Jennifer Nguyen.
TEST PARAMETERS
The test was conducted over the course of three days. Participants were required to complete tasks using specific features of each web-based project management tool. The test was completed at the participant’s home during their free time and also at their office cubicle. After completing the instructions on a Mac or PC, participants were asked to complete the post questionnaires and submit them to the GTU IT test administrator, Valerie Cooper immediately thereafter via email.
USABILITY TEST PLAN
Statement of Purpose
The purpose of the usability test was to identify specific problem areas, inconsistencies, issues, strengths and advantages of Basecamp, Zoho and Smartsheet over other existing web-based project management applications. The users’ reaction and experience with the instructions were measured, observed and examined for the purpose of determining whether or not these applications will be successful in solving the current project management issues and problems involving large scale projects currently plaguing GTU IT Department.
Test Issues
Possible issues with the applications were: unreliability of data security, slow and inefficient file transfer or upload rate, limited storage capacity, no real-time project tracking, no conferencing project management feature, lack of tech support, non-intuitive or confusing website design, applications are difficult to learn or use, very expensive software fees.
Test Goals
The test was designed to evaluate the ease of use of the following features:
• Task management
• Calendar
• Discussion board
• Email notification
• Deadline and Meeting Alerts
• History Tracking
• Templates
Test Environment
Test was performed using a PC, MAC, iPad, iPhone, or Blackberry Devices at the participant’s home or office workstation as outlined below. Participants completed tasks independently of the test facilitator. Participants were encouraged to think aloud, vocalize and tape record their reactions with an audio or video tape recorder as they used the features of the applications.
Day 1: Testing was conducted on site, in the office environment of each respective participant.
Day 2: Testing was conducted in office for Participants 1 and 2, and at remote site Downtown Campus for participants 3 and 4.
Day 3: Testing was conducted in office for Participants 1 and 2, and at home offices of participants 3 and 4.
Recruitment Methodology
Four test participants were screened and selected based on their level of computer use and experience. A wide range of computer skills and job responsibilities were required so that the test may reflect the actual user demographic or typical user profile of the GTU IT Department. Participants recruited had to be employed under Geek Tech University’s IT Department from the main office, as well as at the different field locations.
Every member of the GTU IT staff was contacted via email asking for their voluntary consent to participate in the usability study.
User Profile
Four total participants were selected from the IT department, representative of the members of the department most likely to use the PMT platform:
Participant #1: Part-time employee, minimal computer experience (limited to email and Internet), employed with GTU for six months.
Participant #2: Full-time employee, moderate computer experience (some data entry), employed with GTU for two years.
Participant #3: Full time employee, extensive computer experience (some programming), employed with GTU for five years.
Participant #4: Salaried management, extensive computer experience (graduate degree IT), employed with GTU for seven years.
Participant Recruitment Plan
An email was sent to the entire GTU IT Department asking for volunteers from different teams of discipline and office locations within the GTU IT Dept. Also an informal meeting was conducted at the Main Headquarters to recruit participants. Participating volunteers were given an incentive of $50 gift certificates each towards meals at the GTU Metro Café.
Participant Screening Plan
Each test user had to own a Blackberry, iPad, or iTouch and use both PC and MAC to conduct large-scale projects at GTU IT Dept. Participants also had to be employed at GTU IT Dept. for at least 3 years. Each participant had to belong to different teams of discipline at the GTU IT Dept. and had to be working at the different field locations at GTU IT.
Consent Forms
Participants were asked to sign a consent form via email asking their permission to conduct the test during their free time at their respective homes and during work hours at their cubicles or office. The consent form explained to the participants that the information they provided will be used to measure effectiveness of the web-based project management tools. Participant’s identification would be disclosed but would be used only within the GTU IT Department. The consent form required that the participants complete the test and not withdraw from it.
LIST OF PARTICIPANT TASKS
• Participants read and signed the consent forms and returned them to GTU IT Dept. via email.
• The test administrator asked participants to submit questions they had about the test before they began the test.
• Participants completed a pretest demographic and background info questionnaire.
• For each PMT, Zoho, Basecamp, and Smartsheet, participants were asked to:
1. Add a new job/project or a new “to do list” in the PMT
2. Manage the project in the PMT using the communication or tracking features of the software application and use of message boards or notifications and to change/update the documents.
3. Upload large and complex project files for database storage.
• Participants were asked to record their experience using the features and tools either mentally or by taking notes.
• Participants were asked to record how long they used each application and to describe what the extent of their work was.
• After testing the features of the applications were completed, participants then had to fill out a post-task questionnaire elaborating on their experience.
METHODS OF EVALUATION
Data Collection Forms and Post Test Questionnaires
Questionnaires regarding ease of use and satisfaction were distributed and collected via email using free-form responses and rating scales. The time it took to complete the tasks was also measured, from the time the participants began the tasks until the time the tasks were completed. The participants’ experience was measured against desired usability goals and success requirements, and a tally of errors was kept for inclusion in a frequency report.
TEST RESULTS
Methodology
100% of the participants were able to follow all the tasks assigned to them and completed all tasks as described and listed on the List of Participant Tasks portion of this report. Each participant spent three days using different features of the PMT tools.
The data collected from the questionnaires and the measures obtained, showed that 100% of participants were able to successfully complete their projects tasks with ease and were a great improvement over Geekworks 2.1. 100% of the participants reported that these PMT’s delivered efficiency, kept track of communication and user satisfaction.
Results
BASECAMP
In testing, 85% of the users found navigation of Basecamp to be time-consuming and clumsy. Users were unclear how to use features the first time due to the confusing layout of the screen. Simple tasks, such as adding a new document, were difficult to execute as a result of confusing screen layout. When attempting to add a project or to-do list using a PC, Jaws bugs appeared. Basecamp also lacked some higher-end collaboration features, such as white-boards. Basecamp used significant memory, slowing the application on older PCs, and file-sharing required a separate FTP program. Finally, the writeboard didn’t have a what-you-see-is-what-you-get (WYSIWYG) interface; instead, users had to reference a code sheet for common formatting options.
ZOHO
In testing, 90% of the users described the Zoho’s office applications as too basic and limited. For instance, The Zoho Writer word processor and Zoho Sheet spreadsheet were front ends for outdated versions of OpenOffice.org 2. Zoho writer allowed import of documents in Word 2003 and earlier, but not in the new Word 2007 format. Users couldn’t use features like footnotes or endnotes. The office tools were not fast or powerful enough to deal with large and complex documents, as Zoho bogged down immensely when working with large files. There were no easy links for help or support.
SMARTSHEET
Regarding Smartsheet, 100% of the users were extremely satisfied with the simplicity, ease of use, elegant design and layout of the application. Users found few issues when completing different and large-scale job tasks, as the format of Smartsheet was compatible with all operating systems for both PC and MAC, and could be imbedded in existing enterprise platforms. This allowed users to create tasks and documents with the use of the most current
TEST RESULTS (cont.)
versions of MS Office, or other office applications. The applications were very user-friendly, fast and efficient. Users were able to pass information and documents among employees and teams very clearly and kept projects in one manageable place. There were no major bog-downs during file transfers and uploading. Alerts existed to notify project managers when changes were made to a project, or when the job was completed. There were message boards for communication. Changes could be seen in real time. Documents were easy to locate. Another feature which our users were very happy with was that Smartsheet is Ajax-based and built on commercial-grade open source tools to ensure scalability and reliability. Security features were also impressive, offering privacy settings, access controls, SSL encrypted user authentications and data encryption.
Conclusions
Based on the usability test, Zoho and Basecamp applications offered the most features, but users were overwhelmed with the complexity of applications. In addition, the cost of implementation would exceed the GTU IT budget. A simpler and more efficient project management tool is better suited to the needs of the GTU IT department.
Of the Project Management Tools tested, Smartsheet Project Management Tool proved to be the best solution for the GTU IT department. In addition to ease of use, reliability, and efficiency, Smartsheet is also the most cost-effective solution. Project management is complex, but the simplicity and efficiency offered by Smartsheet is key. In addition, Smartsheet provides remote accessibility, secure data storage and fast downloading, while also serving as an excellent communication standard. In conclusion, we are confident that productivity will be increased with the implementation of Smartsheet as a replacement for Geekworks 2.1.
APPENDICES
Copies of the Consent Form, Data sheets, Screening questionnaires, Pre-test and Post-test questionnaires, and participant responses may be obtained by sending an email to vrcooper@asu.edu or by contacting the GTU IT Department at: Geek Tech University, 135 Main Street, IT Bldg. Room 4, San Francisco CA 94114.
Recent Comments