How do you measure the effectiveness of generative AI training? How to create evaluation sheets and practical assignments

The effectiveness of generative AI training cannot be judged solely by participant satisfaction. We will use tasks that are similar to actual practice and evaluate them, including determining input, creating requests, and checking output. We prepare assignments of the same level of difficulty before and after the training, and check afterwards to see if they were able to use them in actual practice.

Divide the evaluation into three time points

TimeThings to checkMethod
Before trainingCurrent work and tasksPerforming similar tasks using normal means
Immediately after trainingCan you reproduce the procedures you learnedEvaluate your practical skills using different fictitious data
After trial in practiceIs it possible to continue using it?Check the implementation record of the target business

If you use the same questions as they are, your memorization of the questions will be reflected in your grade. We will compare different questions with the same amount of input and difficulty.

Evaluation example using negotiation memo organization

Prepare an assignment that summarizes customer requests, decisions, responsibilities, and deadlines from hypothetical business negotiation notes. Deliberately include items with no deadlines and see if you can notice if guesses are mixed into the AI ​​output.

Perspective0 points1 point2 points
Input judgmentUsing unconfirmed materialsCheck if pointed outCheck availability by yourself
Request detailsPurpose is unknownPurpose is present but conditions are insufficientPurpose, conditions, and output format are clear
AccuracyImportant errors remainCan be corrected and completedCompleted correctly by checking with evidence
Confirmation actionAdopt as isConfirm partConfirm unclear points and basis
ReuseNo steps leftSave only instructionsLeave instructions and confirmation steps

This is an example of grading for training design. Even if your total score is high, if you enter confidential information or make a major error, we will deal with it separately. The total score alone does not determine whether or not it can be used for practical purposes.

Time is measured until completion

This includes input preparation, requests to AI, waiting time, and human verification/correction. Don't compare it to conventional work only based on the time it took to generate. If the instructor helped you along the way, the details of the assistance will also be recorded.

You can use the Evaluation Sheet CSV to record scores for each perspective, time required, and whether support is provided. The scores in the sample entries are fictitious and do not reflect the results of the training.

Items to check after attending the course

Check whether you have had the opportunity to try out the target work, whether you have actually used it, and whether your burden, including confirmation, has changed. If the reason for not being able to use the materials is due to restrictions on the use of materials, organizing the information management may be a better option than additional training.

The difference before and after training is also affected by the amount of work, the experience of the person in charge, and the improvements made at the same time. Do not conclude that all changes are due to training effects; record conditions.

FAQ

Is a survey unnecessary?

This is useful for finding out about parts that are difficult to understand or problems you may have. Satisfaction and practical evaluation are used as separate indicators.

Should I give everyone the same assignment?

Basic input judgments can be standardized, but application to work is tailored to the type of job. We will align the evaluation criteria and adjust the content of the assignments.

Before consulting ZETTAi

Deciding on the work tasks before training and the deliverables you want to check after training will make the evaluation axis clear. Regarding ZETTAi's AI training for corporations, you can consult about evaluation methods that are connected to practical work. The scope of individual support will be confirmed at the time of consultation.

Articles you may also want to read

Information on training and introduction support

If you are the president and would like to try automating your work, please see Contents and prices of AI training for presidents. This is a training program in which we listen to your business and requests, create something, try it out, and make improvements.

Contents and prices of AI training for presidentshttps://ceo-claude-code-kennsyuu.onrender.com/

When considering what to automate with Grokbot, you can specify the work you want to delegate from Grokguide usage examples/prompts/how to make.

Grokguide usage examples/prompts/how to makehttps://grokguide.jp/

If you would like to consult with us about building a system tailored to your business, please check Details of AI agent construction support.

Details of AI agent construction supporthttps://zettai-ai-agent-site-render.onrender.com/