Evaluation of Copilot O365

Copilot is a generative AI tool integrated into the Microsoft ecosystem and was used in the pilot mainly in Outlook, Word, PowerPoint, Excel, Teams and Edge. Participants attended an initial workshop, documented their experiences on a weekly basis and were interviewed after completing the test period.

The results show clear benefits in Word and Outlook in particular. In these programs, participants were able to save time through features such as automatic summarization of email threads, language review, translation and quick rephrasing of text. PowerPoint proved to be a good starter for presentations, although the material was often flat and needed to be adjusted afterwards. Excel, on the other hand, did not meet expectations in terms of data analysis and formula support.

It became clear that the benefits of Copilot are greatest for employees who already have established work habits in the Office tools. For some participants, the tool became more of a distraction when its use was not clear. More than a third reported that they did not have time to test the tool to the extent they would have liked, partly due to lack of time, ideas and uncertainty about how to formulate prompts.

Many participants also had experience using ChatGPT and expressed that the language quality was often better there. At the same time, the strength of Copilot was highlighted as being seamlessly integrated into the work environment and as a reminder to use it.

The pilot study shows that the introduction of Copilot requires more than just access to the tool, it also requires training, changes in working methods and continuous support. A test period of four weeks proved to be too short to establish new routines. There is a clear need for targeted training, for example in the form of practical workshops and ongoing coaching, to ensure that participants feel confident in using the tool.

Generative AI tools are developing rapidly and their functionality is constantly changing. This requires monitoring and updated information prior to any licensing decisions.

In light of this, it is recommended that licenses for Copilot be offered selectively for the time being, i.e. to employees who already work regularly in Microsoft's software and who express a concrete need for the service. The license should be linked to a short training package including an introductory workshop and support.

To handle orders for licenses, a model is proposed where the immediate manager can easily order licenses for employees at the expense of the department/unit via an addition to the existing process. During the autumn, work will be started to review procedures for that process.

For questions about the pilot study or future licenses, contact Maria Nilsson, ICT Educational Developer at Akademus: maria.nilsson@hv.se.