« April 2009 »
1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30
You are not logged in. Log in
Instructional Systems Design - ISD
Assessment Instruments
Topic: 7- Assessment

A- Criterion-referenced instruments
B- Student work samples
C- Student Narratives
D- Performance Standards and Assessment Rubrics

A- Criterion or objective-referenced assessment instruments:
(Learner-Centered) p145
> Assessment that enhances student learning (Baron, 1998)

Criterion-referenced assessments are

  • linked to instructional goals
  • linked to objectives derived from goals

The purpose of this type of assessment is to evaluate:

  1. students' progress: they enable learners to reflect on their own performances > they will become ultimately responsible for the quality of their work
  2. instructional quality: they indicate to the designer which components of the instruction went well and which need revision

+ it contains a criterion that specifies how well a student must perform the skill in order to master the objective.
>Therefore, performance criteria should be congruent with the objectives, learners, and context.

The performance required in the objective must match the performance required in the test. (p 146)

Two uses of criterion-referenced tests:

  • pretests: they have 2 goals
    • to verify that the student possesses the anticipated entry behaviors
    • to measure the student's knowledge of what is to be taught.
  • posttests: they are used primarily to measure the student's knowledge of what was taught.

4 Types of Criterion-Referenced Tests & Their Uses (p.146)

a- Entry behaviors test: It is given to learners before they begin instruction in order to assess those learners' mastery of prerequisite skills.
This entry behavior test should cover the skills that are more questionable than others in terms of being already mastered by the target population.
From entry behaviors test scores designers decide whether learners are ready to begin the instruction.

b- Pretest: It is given to learners before they begin instruction in order to determine whether they have previously mastered some or all of the skills that are to be included in the instruction.
If all the skills have been mastered, then the instruction is not needed.
The pretest includes one or more items for key skills identified in the instructional analysis, including the instructional goal.
Since both entry behaviors tests and pretests are administered prior to instruction, they are often combined into one instrument.
From pretest scores designers decide whether the instruction would be too elementary for the learners and, if not too elementary, how to develop instruction most efficiently for a particular group.
A pretest is valuable only when it is likely that some of the learners will have partial knowledge of the content.

c- Practice/rehearsal tests: They are focused at the lesson rather on the unit level. The purpose for practice tests is to:
- provide active learner participation
- enable learners to rehearse new knowledge and skills
- enable learners to judge for themselves their level of understanding and skill
- enable the professors to provide corrective feedback
- enable the professors to monitor the pace of instruction

d- Posttests: They are administered following instruction, and they are parallel to pretests (excluding entry behaviors items).
Posttests measure objectives included in the instruction. They assess all of the objectives, and especially focus on the terminal objective

Designing a Test (p.149)

A criterion-referenced test is designed by matching the learning domain with an item or assessment task type.

a- Objectives in the Verbal Information Domain: they require objective-style test items which include the following formats:

  • short answer
  • alternative response
  • matching
  • multiple-choice items

b- Objectives in the Intellectual Skills Domain: they require one of the following:

  • objective-style test items
  • the creation of a product (essay, research paper...)
  • a live performance of some type (act in a play, make a presentation, conduct a business meeting...)

If an objective requires the learner to create a unique solution or product, it would be necessary to 

  1. write directions for the learner to follow
  2. establish a set of criteria for judging response quality
  3. convert the criteria into a checklist or rating scale (rubric) that can be used to assess those products

c- Objectives in the Affective/Attitudinal Domain: They are concerned with the learner's attitudes or preferences. Items for attitudinal objectives require one or both of the following:

  • that the learners state their preferences
  • that the instructor observes the learners' behavior and infers their attitudes from their actions

d- Objectives in the Psychomotor Domain: They are usually sets of directions on how to demonstrate the tasks and require the learner to perform a sequence of steps that represents the instructional goal. Criteria for acceptable performances need to be identified and converted into a checklist or rating scale that the instructor uses to indicate whether each step is executed properly.

Writing Test Items (p. 151)

There are 4 categories of criteria that should be considered when creating test items:

  1. Goal-Centered Criteria: The test items and assessment tasks should be congruent with the terminal and performance objectives; they should provide learners with the opportunity to meet the criteria necessary to demonstrate mastery of an objective.

    No rule states that performance criteria should or should not be provided to learners. Sometimes it is necessary for they to know performance criteria and sometimes it is not.

    (Based on my personal experience, here, I would say that it is very beneficial for the learner to receive the criteria before s/he is given the test; this will help them to prepare for it much more efficiently)
  2. Learner-Centered Criteria: The test items and assessment tasks must be tailored to the characteristics and needs of the learners. Criteria in this area include considerations such as learners'
    1. vocabulary and language levels
    2. developmental levels (for setting task complexity)
    3. motivational and interest levels
    4. experiences and backgrounds
    5. special needs

      Designers should consider how to aid learners in becoming evaluators of their own work and performances. Self-evaluation and self-refinement are two of the main goals of all instruction since they can lead to independent learning.
  3. Context-Centered Criteria: The test items and assessment tasks must be as realistic or authentic to the actual performance setting as possible. This criterion helps to ensure transfer of the knowledge and skills from the learning to the performance environment.
  4. Assessment-Centered Criteria: The test items and assessment tasks must include clearly written and parsimonious directions, resource materials, and questions + correct grammar, spelling, and punctuation!

    To help ensure task clarity and to minimize learners' test anxiety, learners should be given all the necessary information to answer a question before they are asked to respond.

    Items should not be written to trick learners! Ideally, learners should err because they do not possess the skill, not because the test item or assessment is convoluted and confusing!

Evaluating Tests and Test Items (p.156)

When writing test directions and test items, the designer should ensure the following:

  1. test directions are clear, simple, and easy to follow
  2. each test item is clear and conveys the intended information
  3. conditions under which responses are made are realistic
  4. the response methods are clear to learners
  5. appropriate space, time, and equipment are available

Using Portfolio Assessments (p. 162)

Portfolios are collections of criterion-referenced assessments that illustrate learners' work. These assessments might include:

  • objective-style tests that demonstrate progress from the pretest to the posttest
  • products that learners developed during instruction
  • live performances
  • assessments of learners' attitudes about the domain studied or the instruction

There are several features of quality portfolio assessment:

  1. the work samples must be anchored to specific instructional goals and performance objectives
  2. the work samples should be the criterion-referenced assessments collected during instruction (the pretests and posttests)
  3. each assessment is accompanied by its rubric with a student's responses evaluated and scored, indicating the strengths and problems within a performance.

The assessment of growth is accomplished at two levels:

  1. learner self-assessment: learners examine their own materials and record their judgments about the strengths and problems + what they might do to improve the materials
  2. instructor assessment: instructors examine the materials set without examining the evaluations by the learner, and record their judgments.

Then both the learner and the instructor compare their evaluations , discussing any discrepancies between the two evaluations.

As a result, they plan together next steps the learner should undertake to improve the quality of his/her work.

B- Student work samples:

  • written reports
  • rough drafts
  • notes
  • revisions
  • projects
  • Journals
  • peer reviews
  • self-evaluations
  • anecdotal records
  • class projects
  • reflective writings
  • artwork; graphics
  • photographs
  • exams
  • computer programs
  • presentations

 C- Student Narratives: A narrative description that

  • states goals
  • describes efforts
  • explains work samples
  • reflects on experience.


D- Performance Standards and Assessment Rubrics:

A variety of scoring criteria can be used in developing performance standards or assessment rubrics. Standards and rubrics should include:

  • performance levels (a range of various criterion levels)
  • descriptors (standards for excellence for specified performance levels)
  • scale (range of values used at each performance level)

Examples of Rubric:

  • analytic assessment rubric
  • holistic assessment rubric: it provides a general score for a compiliton of work samples rather than individual scores for specific work samples

Relevant Links:

Assessment Instruments


Teaching Writing: Approaches & Activities

Posted by Nada at 12:01 AM EDT
Updated: 05/05/09 1:09 AM EDT
Instructional Strategy
Topic: 8- Instructional Strategy

Instructional Strategy involves a huge variety of teaching/learning activities (microstrategies) such as:

  • group discussions
  • independent reading
  • case studies
  • lectures
  • computer simulations
  • worksheets
  • cooperative group projects

Microstrategies: they are pieces of an overall macrostrategy that must take learners from a motivational introduction to a topic learners' through mastery of objectives.

A textbook is a microstrategy that serves primarily as a source of information (and is incomplete instruction).

Macroinstructional strategy: is the complete instruction created by an instructor and involves:

  • defining objectives
  • writing lesson plan and tests
  • motivating learners
  • presenting content
  • engaging students as active participants in learning process
  • administering and scoring assessments (providing feedback)

A well-designed set of instructional materials contains many strategies and procedures.

Three of the major components in the learning process that facilitate learning: (according to psychologists)

  • motivation
  • prerequisite and subordinate skills
  • practice & feedback

Psychologists whose work influences approaches to instructional design:

  • Behaviorists (30 to 40 years ago)
  • Cognitivists (who later modified behaviorists' views)
  • Constructivists (more recent; they suggested new approaches)

The term Instructional Strategy covers the following:

  1. selection of a delivery system
  2. sequencing and grouping content
  3. describing learning components that will be included in instruction
  4. specifying how students will be grouped during instruction
  5. establishing lesson structures
  6. selecting media for delivering instruction

1- Selection of a delivery system: (instruction) (p.184)

The delivery system is the general methodology used for managing and delivering the teaching and learning activities called "instruction."

A delivery system is only part of an overall instructional strategy. Here are some examples of some common delivery systems:

  • traditional model
  • correspondence
  • large-group lecture with small-group Q&A follow-up
  • telecourse by broadcast or videotape
  • two-way, interactive videoconference
  • computer-based instruction
  • Internet or intranet web-based instruction
  • self-paced programs
  • combinations and unique, custom systems

In an ideal instructional design process, one would consider the following:

  • goal
  • learner characteristics
  • learning and performance contexts
  • objectives
  • assessment requirements

 Ideal path for choosing a delivery system:

  1. review instructional analysis and identify logical clusters of objectives that will be taught in appropriate sequences
  2. plan the learning components that will be used in the instruction
  3. choose most effective student groupings
  4. specify effective media and materials that are within the range of cost, convenience, and practicality for the learning context
  5. assign objectives to lessons and consolidate media selections
  6. select or develop a relevant delivery system 

However, in real life, things do not really happen this way. Then, the designer must be flexible and get everything out of the system that it is capable of delivering. The designer must make appropriate adaptations or propose an alternative system.

2- Sequencing and grouping content: (p.186)
A- Sequencing content
What sequence should we follow in presenting content? We should follow our instructional analysis, beginning with the lower-level skills. The instructional sequence tends to be a combination of bottom to top and left to right.
B- Clustering content
We may decide to present information
- on an objective-by-objective basis with intervening activities or
- on several objectives prior to any kind of learner activities
5 factors need to be considered when determining the amount of information to be presented:
  1. age level
  2. complexity of material
  3. type of learning
  4. if the activity can be varied
  5. time required

3- Describing Gagne's 9 learning components that will be included in instruction: (p.189)

Nine events represent external teaching activities that support mental processes of learning (cognitive psychology):

  1. gaining attention
  2. informing learner of objective
  3. stimulating recall of prerequisite learning
  4. presenting stimulus material
  5. providing learning guidance
  6. eliciting performance
  7. providing feedback
  8. assessing performance
  9. enhancing retention and transfer

The purpose for developing an instructional strategy is planning how to guide learners' intellectual processing through the mental states and activities that psychologists have shown will foster learning.

Here are Gagne's events of instruction re-organized into 5 major learning components that are part of an overall instructional strategy:

a- Preinstructional activities
b- Content presentation
c- Learner participation
d- Assessment
e- Follow-through activities


a- Preinstructional activities: 3 factors to consider prior to beginning instruction:

- Motivating learners: John Keller (1987)- ARCS model

  • Attention: gaining and sustaining attention (by using emotional or personal information, asking questions, creating mental challenges, using human-interest examples)
  • Relevance: learners must perceive the instruction as relevant to them; instruction must be related to important goals in their lives > there should be congruence between learners' expectations and the instruction.
  • Confidence: learners must be confident that they can master the objectives for the instruction. If the learners
    • lack confidence > less motivated
    • are overconfident > they will see no need to attend to the instruction

      The challenge is to create the appropriate level of expectation for success (> zone of proximal development)
  • Satisfaction: learners must derive satisfaction (extrinsic rewards or intrinsic feelings of accomplishment?) from the learning experience

 - Informing learners of the objectives > so that they know what to memorize, solve, or interpret.

Providing learners with the objectives helps them to

  • focus their study strategies on these outcomes
  • use more efficient study strategies
  • determine the relevance of the instruction

- Informing learners of prerequisite skills: this will prepare them for the instruction to follow. Two purposes for this component:

  • make sure learners view the relationship between new content and what they already know (can be done through brief test of entry behaviors or by providing learners with a brief description of required entry behaviors)
  • promote learners' active recall of relevant mental contexts in which the new content can be integrated.

b- Content presentation: we should determine what information, concepts, rules, and principles need to be presented. (Avoid presenting too much information, especially that unrelated to the objective)

It is important to

  • define new concepts
  • explain their interrelationship with other concepts

We also need to determine the types and numbers of both examples and nonexamples (deliberate attempt to point out why an example is wrong).

Forms of examples and nonexamples:

  • illustrations
  • diagrams
  • demonstrations
  • model solutions
  • scenarios
  • case studies
  • sample performances

c- Learner participation: Practice with feedback! Learners should be provided an opportunity to practice what we want them to be able to do + they should be provided feedback about their performance.

d- Assessment:

  • Entry behavior tests
  • Pretests
  • Practice tests
  • Posttests 

Refer to the following page: http://www.nadasisland.com/isd/index.blog?topic_id=1112721

e- Follow-through activities:

  • Memory aids for retention (job aids such as checklists are very useful)
  • Transfer of learning: research indicates that learners transfer only some of what they learn to new contexts. The designer must be aware of thatand use every means possible to promote the transfer of learning.
    Instruction is effective if learners can use it to further their study of more advance topics or to perform skills on the job that make a difference in their organization's effectiveness.

4- Specifying how students will be grouped during instruction: (p.207)

The type of student grouping (individual pairs, small group, large group) depends on specific social interaction requirements and is often mixed within and among the learning components in a lesson or unit.

5- Establishing lesson structures: (pp.198-199)

Matching learning components with the amount of guidance needed by the intended learners.

Check Moore & Kearsley's Theory of Transactional Distance (1996)

  • Level of course structure: flexible vs. rigid
  • Level of course dialogue: little interactive communication vs. lots of it
  • Transactional distance: greater vs. lesser
  • Suitability for learner autonomy level: highly autonomous learner vs. less autonomous learner who has not learned how to learn

6- Selecting media for delivering instruction: (p.209)

Media are useful to the extent that they effectively carry required learning components of an instructional strategy.

According to Clark (1983), it is the design of instruction, rather than the medium used to deliver instruction, that determines student learning. 

Detailed Outline for an Instructional Strategy: (p.197)

a- Preinstructional activities

  1. Gain attention and motivate
  2. Describe objectives
  3. Describe and promote recall of prerequisite skills

b- Content presentation

  1. Content
  2. Examples

c- Learner participation

  1. Practice
  2. Feedback

d- Assessment

  1. Entry behavior test
  2. Pretest
  3. Posttest

e- Follow-through activities

  1. Memory aids for retention
  2. Transfer considerations

Educational Activities (Dr. Ryan Watkins)

When you have selected the appropriate instructional strategies for your objectives, it is now the time in the ID process for the designer to consider the educational activities that will be included in the lesson. Each of the activities should promote learner interactions as discussed in last week's readings. A cursory list of potential educational experiences offered by Atsusi Hirumi at the University of Houston Clearlake includes:

listen to a lectureread a journal article
conduct a surveycomplete handouts
conduct library researchparticipate in a debate
assess the work of other studentshandle manipulatives
watch a filmcreate a presentation
analyze current eventsparticipate in a panel discussion
interview othersconduct experiments
visit community resourceinteract with laserdisc
participate in a panel discussionparticipate in a simulation
create or make a graphical presentationwrite research, concept, or position paper
write a reflective paperwatch a demonstration

Active participation of students is essential for the learning process. One challenge, however, is that active learning is not always the easiest instruction to design or implement. Most current education and training programs are based primarily on relatively passive forms of education. And while passive instructional events are appropriate to accomplish some objectives, you will also want to consider some active learning events as well.

Graf and Albright (1994) characterize active learning as:

  • students who are involved in more than listening;
  • less emphasis on transmitting information and more emphasis on developing skills;
  • involvement in higher order thinking skills;
  • engagement in activities;
  • the exploration of learners prior skills, knowledge, attitudes, and abilities.

In creating active learning in your instruction you may want to consider:

  • the development of learning communities;
  • InterActivities (Internet-based activities, webquest, etc.);
  • information gathering activities (information searches, electronic appearances, electronic mentoring, electronic publishing, database creation, etc.);
  • information sharing (keypals, sequential creations, discussion groups, impersonations, generating websites, brainstorming, site surveys, etc.);
  • collaborative problem solving (polled data analysis, parallel problem-solving, simulations, social action plans).

Different Strategies + Lesson Plans (Worth considering, but disregard typing-- mainly punctuation-- mistakes)

Community Language Learning + Lesson Plan

Direct Instruction Model + Lesson Plan (pdf)

Inductive Model + Lesson Plan  (pdf)

Inquiry Model + Lesson Plan  (pdf)

Cooperative Learning + Lesson Plan

Reading- Lesson Plan (under construction)  

Relevant Links:

Example of Instructional Strategy based on ISD

Keller’s ARCS model of motivation

Instructional Design Knowledge Base: Models/Theories

Instructional Design Knowledge Base: Instructional Strategies & Tactics

Design and Sequence Your Way to WBT Interactivity By Atsusi Hirumi and Kathryn Ley

Reigeluth's Elaboration Theory  

Merrill's Instructional Transaction Theory (ITT)

Relevant Books:

A Handbook of Job Aids by Allison Rossett and Jeannette Gautier-Downes (1991)

Electronic Performance Support System by Gloria Gery (1991)

Designing and Developing Electronic Performance Support Systems by L. Brown (1996)

Transfer Of Training: Action-packed Strategies To Ensure High Payoff From Training Investment by Broad and Newstrom (1992)

Distance education: A system's view by Moore and Kearsley (2004)

Posted by Nada at 12:01 AM EDT
Updated: 05/06/09 12:30 AM EDT
Instructional Materials
Topic: 9-Instructional Materials

In individualized instruction, many of the instructional events carried out by the instructor with a group of students are now presented to the individual student through instructional materials.

The instructor's role is different, and even more important than in lockstep instruction. The instructor is still

  • the motivator
  • the counselor
  • the evaluator
  • the decision maker
  • responsible for each student's mastery of the objectives.

The delivery system and media selections: (p.238)

Three factors often cause compromise in media selections and the delivery system:

  1. availability of existing instructional materials
  2. production and implementation constraints
  3. amount of instructor facilitation during instruction

Summary by Susanne Hoepfl-Wellenhofer

When developing instructional materials consider:


1)       The three major components of an instructional package: 

  • Instructional Materials: They contain the content – either written, mediated or facilitated by an instructor (the content includes materials for the major objectives, the terminal objective, and any materials for enhancing memory and transfer).  Instructional materials refer to any preexisting materials that are being incorporated, as well as to those that will be specifically developed for the objectives. The materials may also include information that the learners will use to guide their progress through the instruction.
  • Assessments: All instructional material  should be accompanied by objective tests or by product or performance assessments. These may include a pretest and/or a posttest.
  • Course Management Information: There is often a general description of the total package, typically called the instructor’s manual, which provides the instructor with an overview of the materials. It might include the following:
    • tests and other information considered important for implementing the course.
    • student guidance templates
    • automated class listing
    • student tracking
    • online testing
    • project monitoring
    • grade book
    • a variety of communication and messaging mechanisms

      Special attention should be paid to the ease with which course management information can be used by the instructor or course manager.

2)       The evaluation criteria when selecting existing instructional materials:

a- Goal-Centered Criteria for Evaluating Materials: They are focused on the content of the instruction. Specific criteria in this area include:

  • congruence between content in materials and objectives
  • adequacy of content coverage and completeness
  • authority
  • accuracy
  • currency
  • objectivity

b- Learner-Centered Criteria for Evaluating Materials: They are focused on the appropriateness of instructional materials for the target group. The learner analysis documentation should provide the foundation for this evaluation. Specific criteria in this area include the appropriateness of the materials for the learners with regards to their:

  • vocabulary and language levels
  • developmental, motivation, and interest levels
  • backgrounds and experiences
  • special language or other needs 

c- Learning-Centered Criteria for Evaluating Materials: They are focused on the adequacy of existing materials (do they need to be adapted or enhanced prior to use?).  Materials can be evaluated to determine whether the following items are included and adequate/complete:

  1. preinstructional materials
  2. content sequencing and presentation
  3. student participation and congruent practice exercises
  4. feedback
  5. assessments
  6. follow-through directions for enhancing memory and transfer
  7. delivery system and media formats
  8. learning guidance to move students from one component/activity to the next..

d- Context-Centered Criteria for Evaluating Materials: They are focused on the appropriateness of existing materials for the instructional and performance context. Judge if existing material can be adopted; if not, you are in instructional materials development business. Criteria in this area include:

  • the authenticity of the materials for context and learners
  • the feasibility of the materials for settings and budget. Here examine the technical quality of existing materials with regards to:
    • packaging
    • graphic design and typography

    • durability

    • legibility

    • audio and video quality

    • interface design

    • navigation

    • functionality

A recent development in selecting existing instructional materials is the SCORM: Sharable Content Object Reference Model, which is a set of e-learning standards for interchangeability of learning objects (i.e. lessons or modules).

The theory of SCORM is that cost savings could be realized by distributing learning objects across agencies that teach the same learning outcomes.

The theory is promising and bears watching, but practice currently lags well behind theory. 


3)       Which types of learning components you would like to include:

·         Preinstructional activities (including objectives and review materials + motivational materials and activities)

·         Content (including examples and nonexamples of information, concepts, or skills that need to be learned)

·         Participation activities (for practice) and feedback on students' performance 

·         Assessment of learners’ mastery of new information and skills

·         Activities that enhance memory and transfer


4)       Which types of material you want to include in an instructor’s guide:

·         Information about target population

·         Suggestions on how to adapt materials (for older, younger, higher achieving, or lower achieving students)

·         Content overview

·         Intended learning outcomes

·         Suggestions for using the materials in a certain context or sequence

·         Suggestions for materials management for 

  1. individualized learning
  2. small-group learning
  3. learning-center activities
  4. classroom activities 

·         Retention and transfer activities:

  1. tests that evaluate performance on terminal objectives
  2. evidence of effectiveness of materials
  3. suggestions for evaluating students' work and reporting progress
  4. estimation of time required to use the materials properly
  5. equipment or additional facilities needed for the materials 

5)      If the designer

·         is the developer and the instructor: the whole process of materials development is rather informal.

·         is not the instructor: there might be teams – manager, ID designer, SME (Subject Matter Expert), materials developer and evaluator. Here a premium is placed on precision specifications and working it requires communication and collaboration skills. 

Great summary, Susanne!  The only thing I would add to your summary if I were going to give another designer advice for moving into the development of instructional materials is to have the performance objectives handy.  I found myself constantly refering back to them to keep focus on the conditions I had set and the specific content needed to meet the overall goal.  Also, I would tell them to make sure the materials facilitate the learning (measured by the info gathered in the learner analysis) and that they are conducive to transfer of skills to the actual workplace (or setting where the new skills will be used).   As a last thought, I would add (only because D,C&C mentioned it a few times throughout chapter nine) to remember that the drafts are just that, drafts, so as D,C,&C explain, there is no use in investing a lot of time and money on materials that will most likely be revised and updated (even after they are used in the instruction, but definitely before). 

Patricia Parada

Technical and Instructional Alignment  (By Dr. Ryan Watkins)

The focus of this lesson is on the development of "rough draft materials" for formative evaluation. It is often useful to think of these rough drafts with either "technical" or "instructional" alignment in mind. Let me explain...

If we want to create a rough draft (or prototype) to assess the technical alignment in the formative evaluation, then we want to develop a technical template that may be used throughout the instruction. For example, if we were creating web-based instruction this may be the "home page" that identifies features (e.g., discussion area, online help) that will be available to learners throughout the instruction. In the development of instruction not all of these features have to be fully functioning at the time of the formative evaluation, though at least partially functioning examples should be developed and available to learners during the formative evaluation. Similarly, in text-based instruction items such as a glossary or self-check assessment do not have to be fully developed, but they should be identified in detail within the introduction and/or instructions of the lesson and partially developed for the formative evaluation participants. During the formative evaluation you will assess the technical alignment by considering if these resources are required by learners, can learners access the necessary resources, what other resources may be required, and other related questions.

When we create a rough draft (or prototype) to assess the instructional alignment in the formative evaluation, then we want to develop a functioning module (or selection) of the instruction to be evaluated. In the formative evaluation we will assess this module to determine if it serves as an adequate template for the remainder of the lesson. The module may be related to a discrete objective or to multiple instructional objectives if a variety of instructional strategies were utilized. For web-based instruction this would include all of the educational materials as well as other instructional features available to learners (e.g., glossary of words, online resources, online help related to the module). Likewise, in text-based instruction you will want to develop a portion of your instructional materials to determine if instructional strategies, events, and activities work as desired.

In assessing both technical and instructional alignment you will want to consider issues related to visual literacy and graphic design as well.

Relevant Link:

Example of an Instructor's Package

Example of a Student Guide

Posted by Nada at 12:43 AM EDT
Updated: 05/05/09 1:15 AM EDT
Formative Evaluations
Topic: 91- Formative Evals

In the past, too often were instructors blamed for poor teaching and learners for poor learning when, in fact, the materials were not sufficient to support the instructional effort. This is why formative evaluation to newly developed materials, to selected and adapted materials, to instructor delivered instruction, and to combinations of these 3 presentation modes, is nowadays a must in order to ensure that instruction is properly implemented and managed! 

Formative evaluation is the collection of data and information during the development of instruction that can be used to improve the effectiveness of the instruction (Dick and Carey, p. 277; based on Cronbach, 1975 and Scriven 1967).

=> Formative evaluation is the process designers use to obtain data that can be used to revise their instruction to make it more efficient and effective (gathering information from learners in order to revise the materials before proceeding with the design process).

The emphasis in formative evaluation is on the

  1. collection and analysis of data
  2. revision of the instruction

[Two related activities that share many of the same principles as formative evaluation:  usability testing and rapid prototyping]

Three basic phases of formative evaluation: (p.279)

  1. One-to-one or clinical evaluation: the designer works with individual learners to obtain data to revise the materials
  2. Small-group evaluation: a group of 8 to 20 learners (who represent the target population) study the materials on their own and are tested to collect the required data
  3. Field trial: often 30 learners are sufficient. The emphasis here is on the testing of the procedures

The three phases of formative evaluation are typically preceded by the review of instruction by interested specialists who are not directly involved in the instructional development project, but have relevant expertise.

Designing Formative Evaluations (FEs): (p.279)

The purpose for the formative evaluation is to pinpoint specific errors in the materials in order to correct them.

The best anchor or framework for the design of the FE is the instructional strategy: create a matrix that lists the components of the instructional strategy on one side and the major areas of questions about the instruction on the other.

The 5 areas of questions that would be appropriate for all materials:

  1. Type of learning: are the materials appropriate for the type of learning outcome? (ask expert)
  2. Content: do the materials include adequate instruction on the subordinate skills, and are these skills sequenced and clustered logically? (ask expert)
  3. Clarity: are the materials clear and readily understood by representative members of the target group? (ask target learners)
  4. Motivation: do learners find the materials relevant to their needs and interests? (ask target learners)
  5. Management: can the materials be managed effectively in the manner they are mediated? (ask expert & target learners)

In designing instrumentation for gathering information from learners, you must consider:

  1. the phase: one-to-one, small-group, field trial
  2. the setting: learning or performance context
  3. the nature of the information being gathered

The types of data the we need to collect:

  1. data collected on entry behavior tests, pretests, posttests, and performance context
  2. learners' comments
  3. data collected on attitude questionnaires and/or debriefing comments (reveals learners' overall reactions to instruction)
  4. time required
  5. SME's reactions
  6. manager or supervisor's reactions

Relevant Link:

Example of a Formative Evaluation

Posted by Nada at 12:01 AM EDT
Updated: 05/05/09 1:17 AM EDT
Summative Evaluations
Topic: 92- Summative Evals

Summative evaluations -- which link us back to the needs assessment-- are conducted to document the strengths and weaknesses of instructional materials, to verify their effectiveness with target learners, and to make decisions about whether to maintain currently used materials or adopt materials that have the potential for meeting an organization's defined instructional needs. (pp.340-341)

Summative evaluations answer the questions:

  • "Did the intervention, including the instruction, solve the problem that led to the need for the instruction in the first place?" (p.340)
  • "Did it fill the identified gap?" (Dr. Watkins)

The evaluator is not supposed to be familiar with the materials, the organization requesting the evaluation, or the setting in which the materials are evaluated > external evaluator. 

The 2 main phases of summative evaluation:

  1. Expert Judgment Phase: to determine the potential of candidate instruction for meeting the needs of an organization.
    • Overall decisions: do the materials have the potential for meeting this organization's needs?
    • Specific decisions: (p.341)
      1. Congruence analysis: are the needs and goals of the organization congruent with those in the instruction? > information summary form
      2. Content analysis: are the materials complete, accurate, and current?  > product checklist or rating scale
      3. Design analysis: are the principles of learning, instruction, and motivation clearly evident in the materials?  > product checklist or rating scale
      4. Utility & Feasibility analysis: are the materials convenient, durable, cost-effective, and satisfactory for current users?  > information summary form, product checklist, or rating scale
  2. Field Trial Phase: to document the effectiveness of instruction with target learners in the intended setting.
    • Overall decisions: are the materials effective with target learners in the prescribed setting?
    • Specific decisions: (p.341)
      1. Outcomes analysis: > criterion-reference tests, attitude questionnaires, interviews, observations, company records
        • Impact on learners: are their achievement and motivation satisfactory after instruction?
        • Impact on job: are learners able to transfer information/skills/attitudes to the job setting or to units of related instruction?
        • Impact on organization: are learners' changed behaviors (performance, attitudes) making positive differences in the achievement of the organization's mission and goals?
      2. Management analysis: > questionnaires, interviews, observations, company records
        • Are instructor and manager attitudes satisfactory?
        • Are recommended implementation procedures feasible?
        • Are costs related to time, personnel, equipment, and resources reasonable?

Formative Evaluation Vs. Summative Evaluation:

  • Formative Evaluation: to collect data in order to revise instruction
  • Summative Evaluation: to collect data in order to document the strengths and weaknesses of instruction

Posted by Nada at 12:01 AM EDT
Updated: 05/05/09 12:02 AM EDT
Distance Education, Human Interaction, and ISD
Topic: 93- Distance Educ. & ISD


It goes without saying that distance education is gaining tremendous grounds and is very likely to become The Future of Education. Hence, it needs to be taken very seriously; researchers should team up to find the best ways to develop excellent distance education course materials. The ISD process is in itself an excellent system for the regular classroom. So, one would think that it would be a great idea to adapt it to distance education, just as Young & Young (2002) did. Indeed, their ISD-Web Conversion Process is very comprehensive and can easily be followed for the development of distance education course materials.


The approach to distance education/training that I would follow, based on Young & Young’s article (outlined below), is a combination of online self-study materials that would be very well presented and organized (with media rich simulations), virtual classrooms that “enable live audio AND video-based interactions”, and computer-mediated learning communities “in which learners meet asynchronously in online groups to debate, test, and exchange ideas, as well as receive expert advice.” Besides, since learners in general are so fascinated by Facebook and Twitter, we can make use of one of those two services to promote human interaction. In addition to all the aforementioned, I strongly recommend that the instructor require two class meetings per semester: one towards the beginning and another towards the end. This way, the “human” presence will not be sacrificed.


In order to go into more details, here is how, based on Young & Young’s article, I see the ID process being used for the development of distance education course materials: after re-analyzing the audience, the learning context, the existing materials, and the delivery system, we need to re-design that delivery system, the interface (the overall structure of the Web environment which fosters learners’ information processing—making it user-friendly), the instructional strategies (by cultivating a new view of learning, one that emphasizes how and why, as opposed to what—incorporating a lot of practice and feedback), and assessment (online scoring and security). Afterwards, we need to re-develop materials and make them very interesting and captivating, implement technical deployment and integration and, finally, formatively evaluate to ensure an error-free environment (there should be no errors or shortcomings in how the material is displayed or accessed) and incorporate online evaluations.


As for the instructor/teacher’s role in the design of distance education materials, I firmly believe that it should not be ignored or belittled. Just as no system can replace a mother, no matter how good or less good (not to say bad) she is to her children, no system should replace a teacher or instructor. The human interaction between an instructor and his/her students is invaluable, whether the instructor is good or less good. It is true that some learners have negative experiences with their instructors; it is also true that some children have negative experiences with their mother. Do we remove their mother from their lives?   I am a believer in the instructor’s positive impact on the learners who learn so much more from him/her than just the content of the subject matter. Why should we accept to lose that? So, in order for distance education (based on an adapted version of ISD) to be successful, it needs to give an appropriate role to the teacher/instructor so that the instructor and the students will still have this bond that is developed in regular classrooms.


Nada S.A.


On using the ISD Process for the Development of Distance Education Course Materials:


Based on the article by Andrea & James Young [2002] titled "Converting existing training products for the Web: a new look at the old ISD process" :



There are 4 types of e-learning:

  1. Online self-study materials
  2. Virtual classrooms
  3. Computer-mediated learning communities
  4. Knowledge bases and decision-support tool


  • is time-independent
  • has a more reference-oriented (less didactic) approach to training
  • has a different concept of how, where, and when learning takes place
  • has a different delivery medium: the Web

ISD-Web Conversion Process:

  1.  Re-Analyze:
    • audience
      • Entry-level knowledge and skills (prerequisites)

      • Computer and Internet literacy (Internet access? Same bandwidth?) 

      • Cultural diversity

      • Spoken language(s)

      • New learning application on the job (is it the same for new and original audience members?)

    • learning context
      • Place: where learners will access the training (the office, home, a computer lab)

      • Time: when learners will access the training (during the workday, at night, and/or on weekends)

      • Frequency: will learners use the training for short periods to solve work-related problems, and/or engage with the training for prolonged periods of intense study time?

      • Materials: hardware and software needed to complete the training (do any plug-ins or software applications need to be installed before users can receive the training?)

      • Internet speed: at what speed will users connect to access the training materials?

    • existing materials: not all existing materials can or should be converted for the web. Many interpersonal and problem solving skills associated with management development are difficult to achieve without some face-to-face interaction.

      Decisions about whether or not to convert, and what blend of e-learning solutions to select are influenced by several factors:

      • Types of learning outcomes

      • Budget and time constraints

      • Capabilities of the delivery system (which is supposed to support the delivery method).

    • delivery system: we need to know exactly what the system is or is not capable of before we begin to re-design training for the Web.
  2. Re-Design:
    • delivery system:
      • If you have decided to purchase a commercial delivery system, the features of the system you select will drive many of the delivery system design issues > your task is to choose which features you wish to enable.
      • If you have decided to develop a delivery system from scratch or perhaps customize an existing one > you are faced with a rather large design task. > what attributes and functionality can you afford given your timeline and budget?
    • interface:
      It refers to the overall structure of the Web environment, including patterns of navigation and organization of related groups of information and the graphical presentation and formatting conventions used on individual Web pages. The interface design fosters learners’ information processing.
    • instructional strategies:

      The instructional strategies that worked in the original training will not transfer to the new e-learning environment because modes of interaction, degree of learner control, access to information and learning resources, methods of organizing content, conceptions of time, and technology differ so profoundly in the new environment.
      When making decisions about how to redesign instructional strategies for e-learning, it is helpful to cultivate a new view of learning; one that emphasizes how and why, as opposed to what.

    • assessment:

      When designing assessments to be used in an e-learning environment, there are three primary issues we must address:

      • How to measure learning outcomes online

      • How to score the results of online assessments and provide learner feedback

      • How to maintain a secure online assessment environment

  3. Re-Develop Materials:
    • acquire and re-purpose existing materials:

      Issues to address in the revised copyright agreement include the following:

      • The size of the new audience

      • The use of a password-protected site to limit access to the materials

      • The layout and resolution of the materials in digital format

    • assemble courseware: Reduce file size and use templates
    • enable learning management
  4. Implement:
    • technical deployment: it involves installing the delivery system platform and courseware on servers, configuring the delivery system platform (for example, adding users and courses to the system), addressing firewall and network access issues, and, in some cases, configuring end-user computers to prepare for e-learning.
    • integration: it refers to the human or soft side of implementation.
  5. Formatively Evaluate:
    • quality assurance: an extensive review of the functionality of the course and delivery system is necessary to ensure an error-free environment (there should be no errors or shortcomings in how the material is displayed or accessed). In the software development world, this is referred to as quality assurance (QA).
    • test usability: additional data concerning the usability of the course delivery system should be collected (usability testing).
    • take evaluation online: by using an online form to gather input from learners, you can expect to increase access to geographically dispersed participants.

This retooled ISD process differs from the traditional ISD process in three respects.

  1. First, it envisions the learning as a dynamic and highly personalized event; one that can take place anytime, anywhere, and in any way.
  2. Second, it assumes the use of existing materials and thus has been tailored to meet the challenges of a conversion effort.
  3. Third, it presupposes the Web as the primary delivery media and attempts to take advantage of its unique capabilities to enable superior learning, and ultimately, superior performance on the job.

Relevant Article:


"The Future of Distance Learning: Defining and Sustaining Useful Results" by Kaufman, Watkins, & Guerra (2001)

Posted by Nada at 12:01 AM EDT
Updated: 05/05/09 12:03 AM EDT
Application of ISD

If you are interested in seeing how ISD can be applied:

- Click HERE for a Short Story Unit (by Nada S.A.)

- Click HERE for a Running Training Program (by Carol L.P.)

- Click HERE for a Study Strategy Plan for Reading and Summarizing
  a 200-Page Novel in German (by Susanne H.W.)

Posted by Nada at 12:01 AM EDT
Updated: 05/06/09 9:46 AM EDT

Newer | Latest | Older