Reviewing in Agile Learning Design

The last concept of PODSIR (Plan, Orientate, Design, Select, Iterate, Review) is for the team to review the learning project (prototype or initial iteration) to see if it performs as intended (shown in the blue cycle below).

The first step is to trial it by using representative samples of the target population (the learners). Next, it is evaluated to ensure the goal and objectives that were made in the Planning stage have been met. And lastly, determine what the goals and objectives of the next iteration cycle will consist of (the orange cycle shown below).

The Flow of Agile Learning Design

Trialing

The first part of the Review is to test the training or learning process by performing either a Design Iteration or a Release Iteration (Moggridge, 2007).

Testing a small population of the learners (target population) to perform the learning process or training course is called a Design Iteration— the iteration is performed to test a learning method, function, feature, etc. of the learning platform to a small set of learners to see if it valid.

If you believe the prototype is fully ready or is useful to the learners, but perhaps not fully complete, then you can perform a Release Iteration — the iteration is released as a product to the business unit or customer. Although it may not be fully completed or functional, the designers and customer believe that it is good enough to help the learners to perform better.

For more information, see Validating Instructional Design.

Evaluation

This part of the Review evaluates the results to ensure the goal and objectives that were made in the Planning stage have been met. Evaluation may be defined as the process of gathering information in order to make good decisions. It is broader than testing, and includes both subjective input (opinion) and objective input (fact).

There are at least six major reasons for evaluation, each requiring a different type of evaluation:

Improve the instruction or learning process (formative evaluation)
Promote individual growth and self-evaluation (evaluation is performed by instructor and learners)
Assess the degree of demonstrated performance (summative evaluation)
Diagnose future learning needs (evaluation is performed by the learners, their managers, and instructional designers)
Enhance the learners' sense of merit or worth
Identify or clarify desired performance

In addition, there are five main purposes of evaluation (Bramley, Newby, 1984):

Feedback - Linking learning outcomes to objectives and providing a form of quality control.
Control - Making links from training to organizational activities in order to consider cost effectiveness.
Research - Determining the relationships between learning, training, and the transfer of training to the job.
Intervention - The results of the evaluation influence the context in which it is occurring.
Power games - Manipulating evaluative data for organizational politics (not recommended).

The evaluation determines if your project is complete and may be ended (except for normal maintenance and upgrades), or if the project needs further iterations. If you determine the project is complete, then it normally requires that your customer signs off on it. If it needs another prototype, then perform an iteration.

For information of the various levels of evaluation, see Kirkpatrick's Four Level Evaluation Model.

Plan the Next Iteration

The results of your evaluation determine how much planning is needed in this step. If your evaluation completely failed, then you might need to perform a full Planning and/or Orientation phase again. If everything is going as planned, then you can move on to the Design stage in the next iteration as shown in the flow chart at the top of the page (the orange cycle). If it is somewhere in the middle of the two, then you need to make some adjustments or changes. The chart below can be used as a troubling-shooting heuristic to help pinpoint your problem(s):

Agile trouble-shooting heuristic

Determining if a level of evaluation (right column) was met will tell you the next level to check:

If the desired impact (organizational objective) was met, then the project is going as planned. If not:
- Evaluate the learners' performance. If they cannot perform as planned, then that means either you did not match the correct performance to meet the organization objective or they need to learn other skills that will help them to perform better. If they can't perform, it is either the learning and/or the motivation level:
  - Evaluate what they learned. If they met expectations, then that means that the learning activities did not provide the degree of skill that is required for them to perform. This means you need to improve the learning activities through one or more iterations until the learners can perform to standards.
  - Evaluate their motivation to learn and perform. If they are motivated then check the environment for things that might hamper them, such as not enough time, bad processes, or inefficient work stations. If they are not motivated to learn and perform, have their managers discuss with them the importance of the goals and objectives.

Next Steps

References

Bramley, P., Newby, A.C. (1984). The Evaluation Of Training Part I: Clarifying The Concept. Journal of European & Industrial Training, 8, 6, 10-16.