my blog

What is the relationship between a construct and a measure?

Because an experiment involves examining the relationship between independent variables and changes in one or more dependent variables, defining what is measured—dependent variables—is crucial. The dependent variables are what can be measured and relate to the outcomes described in the research questions.
The research questions are often stated in terms of theoretical constructs, where constructs describe abstract entities that cannot be measured directly. Common constructs in human factors studies include: workload, situation awareness, fatigue, safety, acceptance, trust, and comfort. These constructs cannot be measured directly and the human factors researcher must select variables that can be measured, such as subjective ratings and performance data that are strongly related to these constructs. To assess how smartphones affect driving, the underlying construct might be safety and the measure that relates to safety might be error in lane keeping where the car’s tire crosses a lane boundary. Safety might also be measured by ratings from the drivers indicating how safe they felt. Subjective ratings are often contrasted with objective performance data, such as error rates or response times. The difference between these two classes of measures is important, given that subjective measures are often easier and less expensive to obtain, with a larger sample size. Both objective and subjective measures are useful. For example, in a study of factors that lead to stress disorders in soldiers, objective and subjective indicators of event
stressfulness and social support were predictive of combat stress reaction and later posttraumatic stress disorder. The subjective measure was a stronger predictor than the objective measure [68].
In considering subjective measures, however, what people rate as “preferred” is not always the system feature that supports best performance [69]. For example, people almost always prefer a color display to a monochrome one, even when color undermines performance.
Furthermore, people cannot always predict how they would respond to surprising events in different conditions, like during system failures. Human factors is much more than intuitive judgment (of either the designer OR the participant). It is for this reason that objective data from controlled experiments are needed to go beyond the expert judgments in heuristic evaluations and subjective data.
Subjective and objective dependent variables provide important and complementary information. We often want to measure how causal variables affect several dependent variables at once.
For example, we might want to measure how use of a smartphone affects a number of driving performance variables, including deviations from the lane, reaction time to cars or other objects in front of the vehicle, time to recognize objects in the driver’s peripheral vision, speed, acceleration, and so forth. Using several dependent variables helps triangulate on the truth—if all the variables indicate the same outcome then one can have much greater confidence in that outcome.

P3.28 For an evaluation of a vehicle entertainment system, identify possible dependent variables.

P3.29 What are the benefits of subjective measures?

P3.30 What are the limitations of subjective measures?

P3.31 What is the relationship between a construct and a measure?

Where and how should evidence be obtained? Erika might review crash statistics and police reports, which could reveal that smartphone use is not as prevalent in crashes even though the prevalence of use of these devices for talking, texting, and calling while driving seems high when collected from a self-reported survey. But how reliable and accurate is this evidence? Not every crash report may have a place for the officer to note whether a smartphone was or was not in use, and those drivers completing the survey may not have been entirely truthful about how often they use their phone while driving. Erika’s firm might also perform their own research in a costly driving simulator study, comparing the driving performance of people while the smartphone was and was not in use. But do the conditions in the simulator match those on the highway? On the highway, people choose when they want to talk on the phone. In the simulator, people are asked to talk at specific times. Erika might also review previously conducted research, such as controlled laboratory studies. For example, a laboratory study might show how talking interferes with computerbased “tracking task”, as a way to represent steering a car, and performing a “choice reaction task”, as a way to represent responding to red lights [59]. But are these tracking and choice reaction tasks really like driving? Y No one evaluation method provides a complete answer. These approaches to evaluation represent a sample of methods that human factors engineers can employ to discover “the truth” (or something close to it) about the behavior of people interacting with systems. Human factors engineers use standard methods that have been developed over the years in traditional physical and social sciences. These methods range from the true experiment conducted in highly controlled laboratory environments to less controlled, but more representative, quasi-experiment or descriptive studies in the world. These methods are relevant to both the consulting firm trying to assemble evidence regarding a ban on mobile devices and to designers evaluating whether a system will meet the needs of its intended users. In Chapter 2 we saw that the human factors specialist performs a great deal of informal evaluation during the system design phases. This chapter describes more formal evaluations to assess the match of the system to human capabilities.

be familiar with the range of methods that are available and know which methods are best for specific types of design questions. It is equally important for researchers to understand how practitioners ultimately use their findings. Ideally, this enables a human factors specialist to work in a way that will be useful to design, thus making the results applicable. Selecting an evaluation method that will provide useful information requires that the method be matched to its intended purpose.

3.1 Purpose of Evaluation In Chapter 2 we saw how human factors design occurs in the understand-create-evaluate cycle. Chapter 2 focused on understanding peoples’ needs and characteristics and using that understanding to create prototypes that are refined into the final system through iteration. Central to this iterative process is evaluation. Evaluation identifies opportunities to improve a design so that it serves the needs of people more effectively. Evaluation is both the final step in assessing a design and the first step of the next iteration of the design, where it provides a deeper understanding of what people need and want. Evaluation methods that serve as the first step of the next iteration of the design are termed formative evaluations. Formative evaluations help understand how people use a system and how the system might be improved. Consequently, formative evaluations tend to rely on qualitative measures—general aspects of the interaction that need improvement. Evaluation methods that serve as the final step in assessing a design are termed summative evaluations. Summative evaluations are used to assess whether the system performance meets design requirements and benchmarks. Consequently, summative evaluations tend to rely on quantitative measures—numeric indicators of performance. The distinctions between summative and formative evaluations can be described in terms of three main purposes of evaluation: • Understand how to improve (Formative evaluation): Does the existing product address the real needs of people? Is it used as expected? • Diagnose problems with prototypes (Formative evaluation): How can it be improved? Why did it fail? Why isn’t it good enough? • Verify (Summative evaluation): Does the expected performance meet design requirements? Which system is better? How good is it? Each of these questions might be asked in terms of safety, performance, and satisfaction. For Erika’s analysis, predicting the effect of smartphones on driving safety is most important: how dangerous is talking on a phone while driving?

Table 3.1

Table 3.1 shows the example evaluation techniques for three evaluation purposes. The first rows of this table show methods for understanding and diagnosing problems with qualitative data. Qualitative data are not numerical and include responses to openended questions, such as “what features on the device would you like to see?” or “what were the main problems in operating the device?” Qualitative data also include observations and interviews. These data are particularly useful for diagnosing problems and identifying opportunities for improvement. These opportunities for improvement make qualitative data particularly important in the iterative design process, where the results of a usability test might guide the next iteration of the design. The third row of the table shows methods associated with verifying the performance of the system with quantitative data. Quantitative data include measures of response time, frequency of use, as well as subjective assessments of workload. Quantitative data include any data that can be represented numerically. The table shows that quantitative data are essential for assessing whether a system has met its objectives and if it is ready to be deployed. Quantitative data offer a numeric prediction of whether a system will succeed. In evaluating whether there should be a ban of smartphones, quantitative data might include a prediction of the number of lives saved if a ban were to be adopted. The last two rows show how both quantitative and qualitative data can support understanding people’s needs and characteristics relative to the design. Although methods for understanding (Chapter 2) and methods for evaluation (Chapter 3) are presented in separate chapters, there is substantial overlap between them. In this chapter, we focus on diagnosing design problems and verifying its performance, but evaluations often produce data that can also enhance understanding and guide future designs. Beyond evaluating specific systems or products, human factors specialists also evaluate more general design concepts and develop design principles. Such concept evaluations include assessing the relative strengths of keyboard versus mouse or touchscreen or rotating versus fixed maps. Concept evaluation reflects the basic science that supports the design principles and heuristics that make it possible to guide design without conducting a study for every design decision.

P3.1 How is evaluation related to understanding in the human factors design cycle?

P3.2 What are the three general purposes of evaluation?

P3.3 Would qualitative or quantitative data be more useful in diagnosing why a design is not performing as expected?

P3.4 Would qualitative or quantitative data be more useful in assessing whether a design meets safety and performance requirements?

P3.5 What is the role of quantitative and qualitative data in system design?

P3.6 Why is qualitative data an important part of usability testing?

P3.7 Give examples of qualitative data in evaluating a vehicle entertainment system.

P3.8 Give examples of quantitative data in evaluating a vehicle entertainment system.

P3.9 Describe the role of formative and summative evaluations in design.

P3.10 Identify a method suited to formative evaluation and another more suited to summative evaluation

THREE CASE STUDIES

Three recent interventions are presented briefly
to contrast the project and strategic approaches. First it
should be noted that all three companies and projects
had a high degree of similarity:

1

All were threatening closure of the plant
due to competitive pressures

2

All were in the process of talking about
change in manufacturing, but none had
proceeded as far down this path.

3

In all plants, the active cooperation of top
management and union leadership was a
prerequisite of the project.

4

All were plants in mature industries, i.e.
where technological changes in the product
would be less important to the company
survival than manufacturing prowess.

5

All were large plants, with several hundred
operators

In company. A, an ergonomics program was
started as a project. The aim was to teach operators,
foremen and technical staff how to do ergonomics, both
by classroom training and by undertaking a series of
demonstration projects throughout the plant. Each
project was to be assessed by before-and-after measures
to demonstrate how ergonomics increases both system
performance and operator well being. Over a two-year
period, the project was technically successful in that it
did train many people to become users of task analytic
and job redesign techniques. Success could also be
measured by workplace changes successfully
implemented. However, the project’s non-strategic
aspects were constantly in evidence: operators could not always get released for team meetings, promised
changes were rarely completed on time, direct
intervention by the plant manager was often needed to
insure implementation. To date the ergonomics
program has had minimal impact on the plant's goal of
achieving company-wide top status in quality before the
announced deadline of 199 1.

Company B was tackled in a more strategic
manner. First a team of university personnel worked
with the company to establish strategic needs of the
business. This assessment concerned general
management, strategic planning, sales 8z marketing,
financials, labor relations and manufacturing. Human
factors was a small part of the "manufacturing" area.
From this assessment came a series of immediate needs,
represented by projects which, if completed
successfully, would have a major impact of the business.
One of these projects was operator training, using
human factors techniques of knowledge elicitation to
form the basis of a knowledge and skill training
program. The first part of this system has now been
implemented. Company B cites the University's
intervention as a major reason for deciding to stay in the
region, and to locate its new manufacturing facility here

The case of Company C was in many ways an
intermediate case between the two levels. The company
manufactured precision aircraft parts, using small batch
production by skilled machinists. The brief was to
improve manufacturing quality, again using human
factors techniques of process control analysis.
However, the team included specialists to work with
manufacturing management and labor unions at a high
level as well as at shop-floor level to bring some order
to the a11 too typical chaos of high scrap rates, missed
deadlines and the end-of-the-month shipping crisis.
Ergonomics intervention was centered around a
manufacturing cell, and included a complete system for
floor-level process control based on operator input and
human factors techniques. The presence of this working
cell was a major factor in the decision of another
company to but the plant, expand it, and make it their world headquarters for aviation components. Here the
intervention achieved strategic-level results despite a
lack of prior strategic analysis, compens2ted for to some
extent by the non-ergonomic interventions

CONCLUSIONS

In only one company (A) was the intervention
labeled as "ergonomics" and that had the least impact
upon the company's future. In the other two, it is
doubtful whether either management would classify our
interventions as "ergonomics" or "human factors"
despite the role of these disciplines in helping the
company achieve its strategic goals

Case studies in small numbers can never provide
statistical proof of the superiority of one approach over
another, but they can indicate that manufacturing
success may be achieved by allowing our discipline to
be part of a relatively homogeneous team. We may
have to change the way in which we operate, and lose
some of our separate identity, if we truly wish to impact
manufacturing

How Can Manufacturing Human Factors Help Save a Company:
Intervention at High and Low Levels

Abstract

Now that manufacturing has become a respectable topic in industry, an obvious question is how
human factors/ergonomics can contribute to the improvement of manufacturing. The traditional route
for ergonomics intervention has been a Project route, with a set of objectives agreed between the human
factors engineer and people within the company. Projects, however, do not ask the question of whether
human factors intervention is likely to have an impact on the company's strategic objectives, for
example, remaining in the manufacturing of a particular product

Case studies in a variety of industries are used to conrrast the project approach with a more
strategic approach. It is concluded that the project may represent sub-optimization in that a successful
outcome of the project may have no impact upon company survival without a careful examination of the
strategic plans of the company

MANUFACTURING IS CHANGING

The past ten years have seen the realization that
if the USA is to compete successfully in the world,
manufacturing cannot be neglected. Lessons from more
successful manufacturing nations have been learned by
companies of different sizes, and the results are
beginning to be seen in manufacturing excellence

Perhaps the most fundamental change in the way
leading companies treat manufacturing is to focus more
clearly on the ultimate objectives of the company.
Almost every company of any size now has a "Mission
Statement" with fine words about customer satisfaction
and product quality which, if heeded, would radically
change the way the company thinks. This change is
truly radical as it is forces employees at all levels to
evaluate their decisions against very specific outcomes.
These outcomes rarely include the traditional measures
of monthly output, labor cost variances, minimum first
cost of investment, or machine utilization; measures
which most company employees have known to be the
overriding criteria in day-to-day operations

At higher levels, companies are now undertaking
strategic planning to aid the long-term shaping of the
company. Not every opportunity should be pursued,
only those which fit long-term goals, goals which are
themselves based on an honest assessment of the
strengths and weaknesses of the company. Thus a
company may abandon a traditional market for large
scale mass-production of identical components to
concentrate on manufacturing a customized family of components at lower production volumes. In this way it
can be extremely responsive to customer needs, a key
component of customer satisfaction

Such responsiveness cannot be achieved without
changes in traditional production systems. Quality is
essential: any defect will have an immediate impact
upon output, an impact which cannot be hidden from the
customer by large inventories in a highly responsive
manufacturing environment. Responsiveness also forces
decreased reliance on a multi-level decision-making
hierarchy. Hence the modern emphasis on well-trained
self-organizing small groups (or cells) which are
responsive directly to customer needs. When response
time, quality, and training come to the fore in
manufacturing industry, so must human factors
engineering.

HUMAN FACTORS IN MANUFACTURING

One arena in which the USA is a major power is
in human factors/ergonomics so that it is natural to
consider how we can use this power to improve
manufacturing. Results to date have been less than
spectacular, with major human factors involvement
mainly in nuclear power production (for cognitive and
behavioral interventions) and in a variety of
manufacturing industries at the level of musculo-skeletal
injury reduction. Results in this context means having
an impact on the company's major decisions of which
markets to pursue, where to manufacture goods and how
manufacturing contributes to overall company goals

On a strategic basis, if the USA has a lead in
human factors, that lead should be exploited in
manufacturing. We have had numerous examples from
around the world of how human factors can be a key
element in manufacturing. Harris & Chaney’s book
(1969) was based on a major implementation of
ergonomics within the quality control function of an
aerospace company. On a smaller scale Hasselquist
(1981) showed how redesigning a line on ergonomics
principles had a productivity payback period of less than
half a year, with the added bonuses of a halving of the
error rate and elimination of musculo-skeletal injuries.
The authors suspect that most ergonomists could
produce similar examples from their files.

But the direct impact of human factors on the
manufacturing system which has been changed is really
not the whole question. For example, a program of
ergonomic changes in a shoe manufacturing company
(Drury & Wick, 1984) was particularly successful in
reducing muscular skeletal injuries in the plant, but it
did not prevent eventual closure of the plant as the
parent company responded to foreign competition.
Even a four-year program of ergonomics throughout the
company (which was estimated to have saved over $6
million in productivity increases and injury reduction)
was not enough to prevent closure of the ergonomics
department with the rest of the engineering functions
after a hostile take-over.

Part of the reason for human factors still not
being of central impact on manufacturing is the way in
which we have traditionally intervened. Whether the
ergonomics expertise comes from a group inside the
company or external to the company, the typical
intervention consists of a project. This project is
defined by both the customer and ergonomists in such a
way that both groups are satisfied with the potential
outcome and the intervention methodology.
Unfortunately, the customers and human factors
engineers may have reached a satisfactory
understanding at the wrong level.

Human factors engineers are trained to ask
technical questions (What is your workhest schedule?
How do you extract information from that display?) but
only have a rudimentary idea of the system functioning
beyond this level in manufacturing industry. Thus, ideas
of customer satisfaction with cost, on-time delivery and
quality are addressed, if at all, in terms of cycle times
and error rates. Our customers are often little better. A
safety manager may wish to reduce lost-time injuries, an
R & D manager may wish to estimate performance of a
prototype system or a quality control manager many
wish to reduce human-caused errors. At these levels,
the two parties can talk comfortably, as the connections
between their variables are reasonably obvious. But
what if the customer’s problem has no impact on the
company’s fortunes: Or what if the ergonomist’s time
should be better spent with another project or another
company? We all have a duty to ensure that what may
be a scarce national resource (human factors talent) is
used to maximum advantage.

In contrast the military typically has learned
many years ago that human factors expertise needs to be
continuously available throughout system development
(e.g. Meister, 1971) where it can impact on the broader
aspects of systems effectiveness. Can we make use of
this model in a manufacturing environment where the
higher level customers are (at best) skeptical about
human factors, and the human factors engineers are
untrained in such areas as strategic planning, cost
accounting or employmenflabor policies?

One way in which human factors can maximize
its strategic impact on manufacturing is to be part of a
team which operates at the strategic level. This means
that the team interacts with, and reports to, the highest
levels in management and union so that the strategic
aspects of the intervention are explicitly part of the
project. It does mean however that the intervention will
not be seen as an ergonomics intervention, but as an
indistinguishable part of the overall systems
intervention.

the Job Allocator, a decision support tool based on linear
optimization that suggests to the team leader, for each shift,
the worker-workplace allocations by matching the skills,
knowledge and capacities residing with those required by the
production plan; the tool considers also the personal allocation
preferences defined by the workers (Table 2: O1);

the Training Needs Detector, a tool dedicated to the human
resources management that reasons in the long-run by ponder-
ing the gaps between required skills in the production and those
provided in the actual job allocations in order to identify
persistent skill shortages towards the definition of personalized
training paths (Table 2: O2).

For the validation of the new approach and developed tools, two
lines,oneformicrowaveovensandanotherforfridgeproduction,were
modelled relying on the representation capabilities offered by the
KNOW Platform. The single jobs at each workstation were designed
trying at the same time: (i) to balance the workload by shifting the
tasks among workstations to cope with the upper limit set at 90% of
takt time; (ii) to distribute the tasks with critical requirements, to
balance the cognitive burden and increase the possibility that a pool of
workers can effectivelyand completely match the skill demand; (iii) to
reduce the risk of musculoskeletal disorders due to ergonomically
impactingtasksinsomeworkstations.Joballocationtestswerecarried
outoff-lineinvolvingteamleaderstocomparetheirchoiceswiththose
offered by the linear optimizer

The improved workplace adaptability results in the reduction of
cumulative trauma disorders and psychological stress. The
availability of an all-encompassing digital solution capable to
characterize, from a human-centric perspective, both workers and
production lines allows to integrate all necessary human-related
information and permits to support production facilities improve-
ment in terms of skill-matching, ergonomics and safety. Further-
more, the integration of multi-perspective tools provides
environments that promote awareness of all the human-related
aspects at a glance, thus fostering
first-time-right solutions and
reducing the several design iterations that were necessary before.
Finally, the long-term analysis of mismatch at the boundaries of
human-automation interaction allows to redesign a more humanaware
automation and to promote training activities to
fit workers’
profiles to automation needs.

5. Concluding remarks

In the context of emerging smart factories, the reported analysis
examines the potential cooperation that can bring to a synergic
human-automation solution when the roles are modulated on the
basis of the automation level. The described industrial cases elicit
the current issues experienced within an automation dominated
environment that impacts production systems productivity and
workers’ well-being.

The proposed model for worker-aware adaptive shows how
human well-being drivers are harmonised in the automation
design. According to this model, well-being is achievable only by
implementing adaptability and
flexibility within a sociotechnical
system; automation becomes the keystone of this human-in-theloop
adaptive approach to production. The new factory automation
integrates seamlessly human and digital decision-making by monitoring production performances and workers’ physiological
parameters; this scheme permits the reintroduction of the man-inthe-
loop within factory automation.

This new framework has been validated in the experiments
carried out in two different domains showing respectively how:

a dedicated tool can relief mental workload while operating in
the context of adaptive automation;

an integrated toolset, covering different phases of factory design
and operation, can support human-centric automation.

The proposed automation model can address the identified
gaps. The obtained results prove, qualitatively and quantitatively,
that the integration of the human factors analysis, within
automation design, is a compelling condition for a synergic
improvement of manufacturing performance and human wellbeing
at once. Specifically, the experimental assessment presented
in this paper shows the effectiveness of the proposed model that
permits to overcome the trade-offs of automated manufacturing
environment and humans well-being through a synthesis of the
technical and psychological tools

Future work must employ methods and tools presented in this
paper to face the gaps outlined in Table 2, in different
manufacturing contexts and at any phase of the production
process.

Nowadays the human-automation synergy is fundamental for a
sound development of the innovative production scenarios
according to new emerging paradigms, like Industry 4.0 or Society
5.0. Particularly, the acceleration of scientific and technological
innovations will make the automation role more and more
important and pervasive. Consequently, the future evolution of
manufacturing industry should tackle the challenge to properly
balance the industrial targets with the workers well-being, by
means of human-in-the-loop adaptive automation systems.