123397 Evaluate a learning intervention using given evaluation instruments

PURPOSE OF THE UNIT STANDARD
This unit standard will be useful to evaluators of learning programmes or courses, whether internally or externally.

People credited with this unit standard are capable of:

    Planning and preparing for the evaluation.
    Collecting and recording data.
    Analysing and interpreting data.
    Compiling an evaluation report.
    Reviewing the evaluation process.

LEARNING ASSUMED TO BE IN PLACE AND RECOGNITION OF PRIOR LEARNING
It is assumed that learners have a working knowledge of the NQF, related systems and quality assurance principles.

UNIT STANDARD RANGE
A learning intervention consists of a series of significant learning events that form a coherent whole and lead to the achievement of a significant outcome of learning within the context of the organisational setting. The intervention could include but is not limited to any combination of workshops, focus groups, broadcast, workplace assignments, classroom-based training, training on the job, projects, assessments.

Evaluation is the systematic process of gathering and analysing data and other objective information to determine the quality, value and effectiveness of a learning intervention within the context of the organisational setting with the specific aim of initiating certain actions.

The assessment of candidate-evaluators will only be valid for award of this unit standard if the following requirements are met:
The learning intervention being evaluated leads to a significant, meaningful and coherent outcome of learning that is in line with SAQA’s definition of competence i.e. embraces foundational, practical and reflexive dimensions of competence. Although this is not a rule, for the purposes of this unit standard the learning intervention being evaluated should be of a size that generally requires in the region of 50 – 80 hours of learning time.
The learning intervention being evaluated includes learning of tasks and concepts. Therefore interventions that relate only to highly task-orientated learning and do not demand much, if any, in the way of reflexive competence, will not provide enough scope for the evaluator to prove competence in this unit standard.
The evaluation is to cover at least the following: learning outcomes, learning design, delivery, application of learning, assessment, internal moderation; support services, administration; learning environment; and stakeholder satisfaction. Where possible, the evaluation is to include a consideration of return on training investment and impact in the workplace.

The evaluation may be for interventions using any one of the following modes of delivery:
Distance.
Formal classroom-based training.
On-the job training.

For the purposes of this unit standard, evaluators may use given evaluation instruments and processes. However evaluators will need to be able to adjust the instruments and process for the particular purpose to which they are applied.

Specific Outcomes and Assessment Criteria:

SPECIFIC OUTCOME 1
Plan and prepare for the evaluation.

ASSESSMENT CRITERIA

ASSESSMENT CRITERION 1
The purpose and scope of the evaluation is defined in terms of stakeholder requirements.
ASSESSMENT CRITERION RANGE
Stakeholders – combinations of: provider, client organisations, funders, government agencies, accrediting bodies, learners.

ASSESSMENT CRITERION 2
Evaluation criteria are confirmed as objective, measurable, sufficient and suited to the agreed purpose of the evaluation. The criteria are confirmed to be aligned with the organisational quality assurance system.

ASSESSMENT CRITERION 3
Evaluation instruments selected make use of strategies, methodologies, research methods and information sources that are appropriate to the purpose and context of the evaluation.
ASSESSMENT CRITERION RANGE
Research methods – qualitative and quantitative:
Information sources could include learners, teaching staff, learning support staff, administrative staff and administrative data, learning materials, learner records, course evaluations, facilitator reports, records of results, professional associations.

ASSESSMENT CRITERION 4
Evaluation instruments selected enable valid, sufficient and reliable data to be generated within time and budgetary constraints, and to achieve the purposes of the evaluation.
ASSESSMENT CRITERION RANGE
Instruments can include the following data gathering techniques: questionnaires, interviews, focus groups, observation.

ASSESSMENT CRITERION 5
Evaluation instruments are adjusted to ensure they will generate the data that are required to achieve the evaluation purpose.

ASSESSMENT CRITERION 6
The plan is developed providing details of the purpose and subject of the evaluation, roles and responsibilities of participants, sample sizes, frequency and timing of data gathering, venue/s, logistics, resource requirements and reporting mechanisms.

ASSESSMENT CRITERION 7
The evaluation plan is agreed with relevant stakeholders, and enables the achievement of the evaluation purpose within time and budgetary constraints.

SPECIFIC OUTCOME 2
Collect and record data.

ASSESSMENT CRITERIA

ASSESSMENT CRITERION 1
Data are collected and recorded in accordance with the evaluation plan.
ASSESSMENT CRITERION RANGE
Timing, target groups, tools.

ASSESSMENT CRITERION 2
Data are collected from a representative sample of the target group in accordance with the evaluation plan.
ASSESSMENT CRITERION RANGE
A representative sample contains sufficient numbers and coverage to appropriately reflect the target group, and produce valid and reliable data.

ASSESSMENT CRITERION 3
Data are collected in relation to key quality indicators, including cost, time, outcomes achieved and manageability.

ASSESSMENT CRITERION 4
Data are recorded without errors or omissions.

SPECIFIC OUTCOME 3
Analyse and interpret data.

ASSESSMENT CRITERIA

ASSESSMENT CRITERION 1
The sorting and summarising of data into categories validly reflect the evaluation purpose and serve to focus attention on particular areas.
ASSESSMENT CRITERION RANGE
Learning outcomes, learning design, delivery, assessment, internal moderation; support services, administration; learning environment; application of learning and stakeholder satisfaction.

ASSESSMENT CRITERION 2
Data are displayed in a way that aids analysis and interpretation.

ASSESSMENT CRITERION 3
Identified trends, patterns, and comparisons validly reflect the data.

ASSESSMENT CRITERION 4
Interpretations of data are consistent with the data and are free of bias.

ASSESSMENT CRITERION 5
Findings are unbiased reflections of the data and are supported by evidence.

SPECIFIC OUTCOME 4
Compile an evaluation report.
OUTCOME RANGE
The report would typically include a summary, background information, the evaluation purpose, evaluation methodology, findings, possible solutions and recommendations.

ASSESSMENT CRITERIA

ASSESSMENT CRITERION 1
The summary and presentation of results validly reflect the data and are in a form that is meaningful to recipients.

ASSESSMENT CRITERION 2
Limitations of data interpretation are made explicit.

ASSESSMENT CRITERION 3
Evaluation results are compared with related external data where available so that valid conclusions can be drawn.
ASSESSMENT CRITERION RANGE
External data can include, but are not limited to – previous evaluations, internal and external benchmarks.
Valid means fit for purpose, and relating solely to the attributes or data being evaluated.

ASSESSMENT CRITERION 4
Raw data are made available in an organised form that can be accessed by recipients within the constraints of relevant policy and legislation.

ASSESSMENT CRITERION 5
Recommendations are formulated in consultation with relevant stakeholders, and are relevant to the context, feasible, can be justified in terms of the findings and evaluation purpose, and promote the improvement of learning interventions.

SPECIFIC OUTCOME 5
Review the evaluation process.

ASSESSMENT CRITERIA

ASSESSMENT CRITERION 1
The process is analysed in terms of strengths, weaknesses and fitness for purpose.

ASSESSMENT CRITERION 2
Opportunities and mechanisms to improve upon the process are identified and recorded for integration into future evaluations.

UNIT STANDARD ACCREDITATION AND MODERATION OPTIONS
Assessors for this unit standard must be registered with the relevant ETQA, or with an ETQA that has a Memorandum of Understanding with the appropriate ETQA.
Providers of learning towards this unit standard must be accredited through the relevant ETQA by SAQA, or with an ETQA that has a Memorandum of Understanding with the appropriate ETQA.
Moderators must be registered as assessors with the relevant ETQA, or with an ETQA that has a Memorandum of Understanding with the relevant ETQA.

UNIT STANDARD ESSENTIAL EMBEDDED KNOWLEDGE
The following knowledge is embedded within the unit standard, and will be assessed directly or indirectly through assessment of the specific outcomes in terms of the assessment criteria:
Outcomes-based education, training and development.
Learning styles and methodologies.
Principles of assessment.
Evaluation principles and methods.
The principles and mechanisms of the NQF.
ETQA requirements for programme approval.
Quality Assurance Systems.

UNIT STANDARD DEVELOPMENTAL OUTCOME
N/A

UNIT STANDARD LINKAGES
N/A

Critical Cross-field Outcomes (CCFO):

UNIT STANDARD CCFO IDENTIFYING
Identify and solve problems using critical and creative thinking: planning for contingencies during evaluation.

UNIT STANDARD CCFO ORGANISING
Organize and manage oneself and ones activities: this will be evident throughout the evaluation process.

UNIT STANDARD CCFO COLLECTING
Collect, analyse, organize and critically evaluate information: the whole evaluation depends on careful collection, analysis, organisation and evaluation of information.

UNIT STANDARD CCFO COMMUNICATING
Communicate effectively: this unit standard requires constant communication throughout the evaluation process.

UNIT STANDARD CCFO DEMONSTRATING
Demonstrate the world as a set of related systems: understanding the impact of learning interventions on individuals and organisations.

UNIT STANDARD CCFO CONTRIBUTING
Be culturally and aesthetically sensitive across a range of social contexts: carry out the evaluation in a culturally sensitive way.

TRAINYOUCAN NETWORK ONLINE STATUS