first 3 articles
Journal of Fundamentals
of Mental Health
Mashhad University
of Medical Sciences
Psychiatry and Behavioral Sciences
Research Center
Original Article
Neuropsychological characteristics and theory of mind in ADHD
and normal students
Kobra Abazari1*
; Mohammad Reza Mahdavi2
; Abolfazl Darvishi3
1 Ph.D. in psychology of children with special needs, Instructor of Farhangian University 2 M.A. student of clinical psychology , Azad university of Eslamshahr, Iran
3 B.A in general psychology, Payam-e-Noor University, Baharestan, Iran
Abstract
Introduction: Comprehensive neuropsychological assessment considers symptoms in an individual, based on the
understanding of structure and function of the brain, and can lead to a better understanding of the symptoms, and
eventually more effective treatment recommendations. The aim of this research was to study and compare the
neuropsychological character and theory of mind as well as the correlation between them in ADHD and normal children.
Materials and Methods: This was a casual-comparative study. 25 ADHD children were selected through convenience
sampling and 25 normal children were selected through cluster sampling. Finally, the Connors Neuropsychological
Inventory and Theory of Mind Test were applied to assess the participants. The data were analyzed through multivariate,
t test, and Pearson correlation.
Results: Our findings showed that there is a significant difference between these two groups in terms of the theory of
mind and the three subscales of the neuropsychological inventory (attention problem (P=0.001), executive function
(P=0.0001), and reading function (P=0.027). ADHD children had lower efficiency than normal children did in theory of
mind (P=0.009) and the three subscales of the neuropsychological inventory. However, there was no correlation between
neuropsychological characteristics and theory of mind in ADHD and normal children.
Conclusion: It can be concluded that neuropsychological tests can be used as a complement to behavioral tests for
diagnostic assessment of ADHD children. In addition, it was observed that ADHD children require rehabilitations
program focused on academic performance, attention problems, and reading performance, as well as theory of mind.
Hence, the necessary measures should be taken for these children.
Keywords: Attention deficit hyperactive disorder, Neuropsychologic, Theory of mind
Please cite this paper as:
Abazari K, Mahdavi MR, Darvishi A. Neuropsychological characteristics and theory of mind in ADHD and normal students. Journal
of Fundamentals of Mental Health 2017 Jan-Feb; 19(1): 22-9.
Introduction
Attention deficit hyperactivity disorder (ADHD), a
neuro-developmental disorder, is characterized by
three main features including attention deficit,
hyperactivity and impulsivity, which it affects on 3
to 7 percent of children (1). Some of the evidence
declares that the physical abnormalities are the
reason to prevalence of the attention deficithyperactivity behavior (2,3), As a result, the disorder
is conceptualized as Neurodevelopment disorder (4-
8).
Partial damage to the brain, which has mentioned as
the cause of hyperactivity, is not recognizable
because of the fallibility of neurological tests (9).
The studies based on Neuroimaging have raised the
conflict between the sub-cortical and thalamocortical processes in frontal networks. In recent
decades, influenced by these findings, the
neuropsychological pattern related to attention deficit
hyperactivity disorder (ADHD) was proposed by
neuropsychology. The cognitive deficits, exclusive
damage in attention and executive function are the
main hypothesis proposed in the scope of this
disorder. The children suffering ADHD gain the
lower scores and exhibit a weaker performance in
multiple tasks such as vigilance, sustained attention
and motor inhibition, executive function, verbal
learning and memory (10-15). It is reported that the
deficit pattern is similar to the findings obtained
from the frontal damaged individual and it has
considered as the basis of the frontal cortex damage
hypothesis or ADHD (10,16).
*Corresponding Author: Isfahan University, Isfahan, Iran
kabazari@yahoo.com
Received: Apr. 13, 2016
Accepted: Sep. 24, 2016
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 22
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 23
Diagnostic criteria, which are based on behavioral
symptoms, cannot sufficiently describe the nature of
some childhood disorders such as ADHD. In some
cases, these children exhibit the severe destruction
in operating function as well as deficits in motor
control and emotional regulation. The
comprehensive neuropsychological assessment,
which considers the available symptoms in
individuals and is based on the understanding the
structure and function of the brain, is led to greater
understanding of the symptoms, and eventually,
more effective treatment recommendations (17). In
addition, the available findings about the different
functional and structural development of ADHD
children’s brain indicates that several nervous
systems are involved in cognitive and motor
disorders as well as the emotional-behavioral
symptoms of the ADHD children. Thus, if the
comprehensive neuropsychological assessment does
not implement, some important condition, that there
is simultaneously in ADHD children, may not be
assessed (17). The new findings support the use of
neuropsychological tests for the differential
diagnosis between ADHD and normal individuals.
Although, the controversial findings have observed
in studies conducted on the normal and ADHD
individuals (13). For example, previous studies have
reported that there is no difference among the
normal and ADHD individual in case of
neuropsychological variable of visual attention (18).
Therefore, these studies show the importance of
neuropsychological tests to assess and clarify the
strengths and weaknesses of the attention in
individual, especially the recognition, and ultimately
to identify the ADHD individuals. Furthermore, the
studies suggest that the neuropsychological test to
diagnose the ADHD in individual is more important
than the use of the only measurement of individual
based on the evaluation tools such as DSM criteria
(19, 20). In addition, it is possible that the deficit in
executive function and neurological problems of
ADHD children such as attention and memory be
due to the deficit of their theory of mind (21).
Theory of mind, or the ability to ascribe the mental
states to oneself and other, is considered as an
outstanding achievement in human development.
Theory of mind allows us to consider the thoughts
and feelings of others beyond our own first-person
perspective. Also, this feature allows us to prepare
for participation in complex social interactions, yet
convenient (22). Previous studies have indicated that
ADHD children are lower than normal children in
theory of mind are (23-28).
However, the development of theory of mind is
initiated by rudimentary skills including attention,
the use of mental condition and pretend play in the
early stages of children's life. The recent studies
have suggested a close link between executive
functions and theory of mind. The researchers claim
that the carrying out the task of theory of mind
significantly requires the executive functions
(9,23,29-31). The results imply that there is the
fundamental involvement of executive functions in
the development of theory of mind. This view is
consistent with the findings of other researchers
(22,24).
Generally, ADHD and the disorders associated
with ADHD lead to difficulty of the definition,
assessment and treatment of this disorder.
Therefore, it is important to consider the wide range
of factors and conditions that cause the inattention,
hyperactivity, impulsivity (32). In addition, there are
some contradictions in the neuropsychological
features of the ADHD individual. Thus, this study
is necessary to detect the disorder in school-age
children, considering the neuropsychological
features and theory of mind of the ADHD children
compared with normal children. Hence, the aim of
this study was to evaluate the neuropsychological
features and theory of mind in children with ADHD
compared with normal children. In addition, this
study was to clarify whether the neurological
problems are related to the theory of mind of ADHD
children.
Materials and Methods
This study, based on its purpose and application, is
a casual-comparative, which is one of the
descriptive (non-experimental) research methods.
The statistical population of this study had been
consisted of the normal and hyperactive boys of
Eslamshahr city. The research was conducted with
the permission of the Administration of Education
of Eslamshahr city.
The sample of ADHD children was selected from
available ADHD boys (age group of 7-12 year old)
in the Razi Counseling Center of Eslamshahr. All
of the test subjects have expressed their contest
towards the participation in the test. It has also
emphasized that the information will be
confidential. The CSI4 test (parent's form) was
implemented to complete the diagnosis of ADHD in
children who were previously diagnosed as the
hyperactive by a psychiatrist. Finally, a number of
37 children have selected as the initial samples. The
inclusion criteria of the subjects in this study were
the normal intelligence range obtained through the
Stanford-Binet intelligence test that was conducted
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 24
on 37 hyperactive children. A number of seven
individual had average intelligence lesser than 90
and they excluded from the study due to this
problem. Finally, the children with IQ range of 90-
120 were included in this study. Of other exclusion
criteria, in addition to IQ 90, was the presence of
the symptoms of autism as well as other associated
disorders, i.e., anxiety, depression and epilepsy and
seizure problems. The observation of the child and
parents’ interview indicated that two children had
the autism symptoms, therefore they has excluded
from the study. There were no epilepsy symptoms in
37 selected children but three of them had anxiety
disorders. Eventually, 25 boys in the age group of 7-
12 years old were included into the study. In order
to assess the associated disorders (autism, anxiety
and depression), CSI4 questionnaire was used. The
children were referred to a psychiatrist for the
implementation of the EEG to measure the
symptoms of children with epilepsy and seizure in
addition to background checks and interviews with
parents and family of the child.
A list of boys’ ordinary schools at the elementary
level at Eslamshahr city were provided to achieve
the groups of children that can provide suitable
statistical analysis, i.e., age and gender appropriate
to the requirements of this study. Then, the boy’s
school of Shohdayeh 17 Shahrivar was selected by
the cluster sampling. After achieving the permission
to attend and research in that school, the list of
students, who had not notable disorder based on the
verification of school counselor, was given and we
randomly selected 25 students to participate in the
study. The questionnaire of ADHD and associated
disorders in this group was implemented based on
CSI4 test (the parent form). According the results of
this questionnaire, the number of 2 children had the
anxiety symptoms and one of children had the
depression symptoms. These three children were
excluded from study, three other normal students
were randomly replaced, and CSI4 test were
performed. Since the teachers and academic and
administrative staff confirmed that these students
have good learning situation, the Intelligence Test
was not conducted. The results indicated that there
was no disorder in these students. Finally, the
studied students along with their parents responded
to the questions of neurological questionnaire and
theory of mind tests.
Research instruments
In this study, four tests were utilized that include:
- Theory of Mind Test (TOM TEST): The main
form of theory of mind test has been designed in
order to assess the theory of mind in normal children
and children with pervasive developmental disorders
at the ages of 5 to 12 years. It provides the
information about the extent of social perception,
sensitivity and insight of children. In addition, it
clarifies the rate and degree to which the children
are able to accept the feelings and thoughts of others
(33). The reliability of the test by Cronbach's alpha
was obtained to be 0.92, 0.84, 0.86 and 0.85 for the
total scale, first scale, second scale and third scale,
respectively (33).
Qmrany et al. created some changes in this test.
They reduced the number of questions of the test
from 72 to 38 and used the Persian name instead of
the foreign names. Then, they measured the validity
and reliability of the test on a number of normal and
educable mental retarded (mild) students at Shiraz.
The content validity method, simultaneous validity
and correlation of subscale with total score were
used to evaluate the validity of the test.
Simultaneous validity was estimated to be 89%
through the correlation of test with the Dollhouse
task, which it was significant at the level of 0.01.
The correlation coefficient of the subtests with the
total scores was significant in all case and it was in
range of 0.92 to 0.98. Test-retest reliability was
between 70% and 96% and the entire coefficient
was significant at the level of 0.01. Internal stability
of the test (using Cronbach's alpha) for the whole
test and each of subtests was calculated to be 86%,
72%, 80% and 81%, respectively.
This test has designed based on a multidimensional and evolutionary perspective of the
theory of mind (33) and it is able to assess the
greater range of age groups and more sophisticated
and complex levels of theory of mind rather than the
older tests such as dollhouse task, boxes of Smartiz,
etc. The main form of the test consists of 78
questions and 3 subtests, which are as follows:
First subscale: Preliminary theory of
mind, i.e., the first level of theory of mind
or recognition of emotions and pretend,
consisting of 20 questions.
Second subscale: Initial statements of an
actual theory of mind, i.e., the second level
of theory of mind or initial false belief and
understanding of false belief, consisting of
13 questions.
Third subscale: Advanced concepts of
theory of mind, i.e, the third level of
theory of mind or secondary false belief
and understanding of joke, consisting of
five questions.
Participants can receive scores between 0 and 20 in
first subtest; scores between 0 and 13 in second
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 25
subscale; scores between 0 and 5 in the third
subscale; and scores between 0 and 38 in the total
score. The sum of the scores obtained from above
three subscales gives an overall score for theory of
mind. Higher scores indicate that the child has
achieved a higher level of theory of mind (33).
- Connors Neuropsychological Inventory (Parents
Form): Connors designed this test to assess the
neuro-cognitive problems with children in age range
of 5-12 years. This test assesses the problems of
attention, sensory-motor function, language,
executive functions, memory, learning and cognition
in four spectrums (unobserved to severe). Jadidi and
Abedi have translated and standardized this
questionnaire and they have obtained its validity
using the factor analysis method. They have
reported that this tool has appropriate validity. The
reliability of this tool using Cronbach's Alpha has
reported to be 0.72. The reliability of subscales was
also determined using Cronbach's Alpha and it was
observed that the reliability of various variables
including executive function, attention, sensorymotor function, language and memory & learning
was 0.71, 0.74, 0.78, 0.69 and 0.79, respectively
(35).
- The Stanford-Binet Intelligence Test: AFrooz and
Kamkari developed the fifth Edition of StanfordBinet in 2008. It is utilized to test IQ for the age
range of 2-90 years old and it can be used in the
areas of identification, diagnosis and placement of
individuals in special education programs. The
profile has concentrated to identify the detailed
performance of the individuals in the 10 subtests
with emphasis on two verbal and non-verbal areas,
which it is consisted of 10 subtests according to 5-
intelligence factor. These 10 subtests in verbal and
nonverbal scale include fluid reasoning, knowledge,
quantitative reasoning, visual-spatial processing and
working memory. In addition, the correlation
between the two areas of non-verbal and verbal
obtained to be between 0.94 and 0.97. The validity
coefficients of this test were extracting to be
between 0.84 and 0.89 among the 10 subscales this
intelligence test. The coefficients calculated for the
tool represents that the tool has high credit for
subtest and combined scores (36). In Iran, The
validity of this test along with Wechsler for the
verbal IQ, non-verbal IQ, and general IQ was 0.58,
0.59 and 0.66, respectively (37).
- Child Symptom Inventory-4 (CSI4): The last
edition of this questionnaire has two forms for
parents and teachers that it shows its suitable
effectiveness compared to other measures and
practices. Furthermore, the diagnostic criteria of this
test are Diagnostic and Statistical Manual (DSM) of
Mental Disorders criteria and its terms and phrases
are the simple and understandable. Two have
designed for scoring methods for Child Symptom
Inventory: Scoring methods based on cut off point
screening and the scoring method based on the
severity of symptoms. In the screening method, the
method of scoring is obtained by sum of the number
of phrases that which are responded by the options
"sometimes" or "often". If the answer of the
questions of the test were the "never" or "rarely", the
"zero" is given to reply to those questions, while the
score of "1" is given to the answers of "sometimes"
or "more often". In scoring by severity of symptoms
method, the options of "never", "rarely",
"sometimes", "more often" is scored with the codes
of "0", "1", "2" and "3", respectively and the sum of
the obtained scores give the symptom severity.
Sprafkin and Gadow et al. (2001) have investigated
the reliability and validity of the Child Symptom
Inventory (CSI_4) questionnaire. The results of this
study have showed that there was satisfactory
internal consistency reliability, test-retest reliability
as well as stability during a period of 4 years. The
CSI-4 scale had convergent validity with the Child
Behavior Checklist (CBCL) and children diagnostic
interview. In addition, the CSI-4 scale had had
divergent validity with parents modified form
(DICA-P) (38). Ismaili & Alipour obtained the
reliability and validity of this test for children at
Tehran at 2002. The validity of this test for
abnormalities was achieved as follows: ADHD 60%,
ADHD predominantly inattentive type (ADHD-PI)
0.53%, predominantly hyperactive-impulsive type
0.69%, Only hyperactivity 0.70, depression 0.56,
Anxiety 0.62, autism 0.58 (39).
Results
The statistical analysis results showed that 11 years
old children have the highest frequency in this
study. Generally, the mean age of the study was
9.26 years. In addition, the 11-year-old children had
the highest frequency in the normal group of
children in that the mean age of the normal group
was 9.73 years and its variance is equal with 3.21.
The standard deviation of this group is 1.79. Among
the ADHD group, 11-year-old children had also the
highest frequency. The mean of age and its variance
are 8.8 and 4.45, respectively. The standard
deviation (SD) of this group was calculated to be
2.11.
Table1 shows the descriptive scores of hyperactive
and normal children in neurological variables.
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 26
Table 1. Mean and SD of the subscales of Connors
test among subjects
Subscales Status Mean SD
1 Attention Problems Normal 16.8 10.15
Hyperactive 29.06 8.15
2 Sensori-motor Function Normal 1.86 3.06
Hyperactive 5.53 14.40
3 Language Functions Normal 3.2 5.28
Hyperactive 5.6 5.81
4 Learning and Memory
Function
Normal 7.13 9.47
Hyperactive 14.06 11.84
5 Executive Functions Normal 6.86 6.55
Hyperactive 18.86 9.63
6 Ability and Speed of
Cognitive Processing
Normal 3.4 4.67
Hyperactive 6.33 6.27
7 Visual and Spatial
Performance
Normal 1.26 2.15
Hyperactive 2.66 3.06
8 Academic Performance
(Reading)
Normal 4.33 6.22
Hyperactive 10.93 9.01
9 Academic Performance
(Writing )
Normal 9.2 8.87
Hyperactive 13.8 15.21
10 Academic
Performance(Math)
Normal 5.2 7.07
Hyperactive 5.66 5.97
The Multivariate analysis of variance (MANOVA)
test was used to compare the differences between
the scores of ADHD and normal individual obtained
from the subscales of Connors neuropsychological
test and the results were represented following table
(Table 2).
Table 2. Differences between hyperactive and
normal groups of children in neuropsychological
variables
Variables
Degrees of
freedom (df)
Mean of
square
F ratio
Significance
level
Efficiency
coefficient
Attention Problems 1 1128.533 13.303 0.001 0.322
Sensorimotor Function 1 100.833 0.93 0.343 0.032
Language Functions 1 43.200 1.4 0.247 0.048
Learning and Memory
Function
1 360.533 3.136 0.087 0.101
Executive Functions 1 1080 15.904 0.0001 0.362
Ability and Speed of
Cognitive Processing
1 64.533 2.109 0.158 0.070
Visual and Spatial
Performance
1 14.7 2.097 0.159 0.070
Academic Performance
(Reading)
1 326.7 5.444 0.027 0.163
Academic Performance
(Writing )
1 158.7 1.023 0.321 0.035
Academic Performance
(Math)
1 1.633 0.038 0.847 0.001
Multivariate analysis (MANOVA) was used to
show the difference between the normal and ADHD
groups of students; Wilks's Lambda with significant
level of 0.0001 represents the difference between
normal and ADHD children in neuropsychological
features.
As can be seen in Table 2, ADHD children had
more problems in three subscale of children's
Connors neuropsychological test including
“attention problems”, “executive functions” and
“academic performance in reading" than normal
children.
In the theory of mind test, the mean of normal
children (SD=3.08) was obtained to be 30.6 while it
was 25.53 for the ADHD children (SD=6.03). In
order to understand the significant differences
between the two groups of children in test of theory
of mind, the independent t-test was used and the
results indicated that there is significant difference
between two groups for the theory of mind test
(t=2.77, df=28 and significant level=0.009). Based
on this result, the group of ADHD children has
lower score in theory of mind than normal children.
In addition, the Pearson correlation test was
applied to measure the correlation level of the
Connors neuropsychological tests and theory of
mind for both of normal and ADHD children. The
results showed that the correlation between theory
of mind and the scores obtained from Connors
neuropsychological test for normal children is as r=-
0.338, p=0.228, there was no significant correlation
at the level of 0.05. The measurement of correlation
between theory of mind and the scores obtained
from Connors neuropsychological test was also
conducted for ADHD children and the results was
r= -0.105, p= 0.701; there was no significant
correlation at the level of 0.05
Discussion
This study aimed to evaluate the neurological
characteristics and theory of mind in ADHD and
normal children at the age of 7-12 years. The results
indicate that the ADHD children were significantly
different from normal children solely in the scale of
attention, executive functions and reading among
the 10 neurological subscales; it means that they had
lower performance in these three scales. The results
of the study are in accordance with results of some
other researchers. They believe that the ADHD and
healthy individual have statistically significant
difference in one or several tests of executive
functions. Approximately, 80 percent of ADHD
children have deficits, in at least one of the
components of executive function (10-14). Also,
Perner et al. express that ADHD children has not
good performance in activities that are often
required to selective attention and executive skills,
as well as sustained attention and executive skills, or
both (15).
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 27
As mentioned in the results, hyperactive children
are also weak in reading performance, in addition to
attention and executive functions, than normal
children. The researchers introduce the attention as
important factor for reading ability, which their
expression can be a proper justification to the
mentioned result. It has also reported that the
attention is important to convert the print into
speech. Therefore, the "attention" is necessary for
fluent reading (29). Some researchers concluded that
deficits of attention is caused the problems in a
person's reading (15, 19,20). Gatica has stated that
there is a relationship between the maintenance of
attention and reading without the loud. People with
good maintenance of attention have faster reading
ability than those with medium and low
maintenance. Therefore, the high ability to
maintenance the attention is leading to the
successful reading comprehension (21). Thus, in
present study, because the hyperactive children
received a low score on attention variable, they had
low scores in reading.
In the next part of this research, theory of mind
were compared between ADHD and normal
children and the results showed that ADHD children
have lower scores on theory of mind compared to
normal children. The results of present study are
confirmed by the previous studies (24-30).
Hughes and Ensor have been studied the theory of
mind among the primary school children with the
behavioral problems; In a sample of 130 children, it
was found that deficits in theory of mind task is a
strong predictor of behavior problems such as
conduct disorder, oppositional defiant disorder and
hyperactivity (20). Walker has been used the false
belief and showed that the theory of mind, after
controlling for age, can significantly predicted
aggression and disruptive behavior in boys (32).
Buitelaar et al. reported that ADHD children in the
false belief are considerably lower than the normal
children are (26).
Some researcher proposed that theory of mind
explains to parents the reason of externalizing
behavioral problems and understanding the mental
states and considering the opinions of others their
child (40). In addition, some theoreticians purpose
that failure in identification of emotions is observed
in children with externalizing problems (41).
Theoreticians of cognitive models believe that
various cognition such as beliefs, attributions and
expectations of yourself and others behavior are
very important in determination of feelings and
behavior; and individual, who have distorted,
inaccurate and inconsistent cognition about
themselves, others and their environment event,
exhibit problematic behavior and feelings.
Aggressive children are also in trouble in front of
peers and it is leading to show lesser empathy in
interaction with peers; and consequently, they will
rejected by their peer that it has significant
relationship with deficits of the of the theory of
mind task (42).
Of other results of the present study is that there is
no relationship among the neurological
characteristics and theory of mind in both ADHD
and normal children. Although, the studies, the
recently published papers about the relationship
between executive function and theory of mind,
have suggested a close link between executive
functions and theory of mind. The researchers claim
that the task of theory of mind is significantly
required to executive functions (9).
It is believed that their findings are statistically
shown the significant relationship between
inhibitory control and task of theory of mind. These
findings highlight the fundamental involvement of
executive functions in the development of theory of
mind. This view is consistent with the findings of
other researchers (30, 31). The studies show their
relationship between executive function in
preschool, which are externalized behavior
(attention, memory, behavioral inhibition, control or
impulsivity, self-regulation) and ideas of theory of
mind or emotions, emotion of their theory of mind
(29).
Of the limitations of this study was the use of only
one gender i.e. boys; thus, it is suggested that the
future studies conducted on both girls and boys and
the neuropsychological features to be investigate
and compared among both genders. Another
limitation is the use of only one tool to measure
neuropsychological features. It is recommended that
other tools be used to assess the neuropsychological
features of these types of children in future studies.
This test was only conducted in Eslamshahr and
children of this area, which it can be considered as
another limitation of this study. The next limitation
can be relatively small sample size, which it is due
to low availability of these children in centers. Since
the results of this study showed the poor
performance of ADHD children in educational
performance of reading, attention and executive
functions, and, since the results are consistent with
other studies conducted in other countries; therefore,
it is suggested that these three components be
applied in training and rehabilitation of ADHD
children. Furthermore, it is recommended that the
questionnaires and the neuropsychological
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 28
diagnostic tool be used to comprehensively assess
and detect the hyperactive children in addition to
interviews and observation methods,. It is also
recommended that the study be conducted for other
childhood psychiatric disorders and for children
with special needs such as learning disorders,
autism, and mental retardation. Another suggestion
is that this study be tested through other
neuropsychological questionnaires as well as other
psychiatric tools, which measure the
neuropsychological features. In addition, this study
can be carried out separately or comparatively in
various groups of the ADHD children (inattentive
type, hyperactive type, composition type). Finally, it
is recommended that this study be performed with
higher sample size and in other regions.
Conclusion
It can be concluded that neuropsychological tests
can be used as a complement to behavioral tests for
diagnostic assessment of ADHD children. In
addition, it was observed that ADHD children
require rehabilitations program focused on academic
performance, attention problems, and reading
performance, as well as theory of mind. Hence, the
necessary measures should be taken for these
children.
Acknowledgments
This research was conducted with the approval of
education administration of Eslamshahr city.
Hereby, we are grateful from the Razi Counseling
Center of Eslamshahr city to cooperate in this
research. In addition, there was no conflict of
interest between the authors.
References
1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. Washington, DC:
American Psychiatric Association. 4th ed; 2000.
2. Barry RJ, Johnstone SJ, Clarke AR. A review of electrophysiology in attention-deficit/hyperactivity disorder: II.
Event related potentials. Clin Neurophysiol 2003; 114(2): 184-98.
3. Jonkman LM, Kenemans JL, Kemner C, Verbaten MN, van Engeland H. Dipole source localization of event related
brain activity indicative of an early visual selective attention deficit in ADHD children. Clin Neurophysiol 2004;
115(7): 1537-49.
4. Castellanos EX, Glaser PEA, Gerhardt GA. Towards a neuroscience of attention-deficit/hyperactivity disorder:
Fractionating the phenotype. J Neurosci Methods 2006; 151(1): 1-4.
5. Rauch SL. Neuroimaging and attention-deficit/hyperactivity disorder in the 21st century: What to consider and how
to proceed. Biol Psychiatry 2005; 57(11): 1261-2.
6. Sonuga-Barke EJ. Causal models of attention-deficit/hyperactivity disorder: From common simple deficits to
multiple developmental pathways. Biological Psychiatry 2005; 57(11): 1231-8.
7. Swanson J, Castellanos FX, Murias M, LaHoste G, Kennedy J. Cognitive neuroscience of attentiondeficit/hyperactivity disorder and hyperkinetic disorder. Curr Opin Neurobiol 1998; 8(2): 263-71.
8. Taylor E. Developmental neuropsychopathology of attention deficit and impulsiveness. Dev Psychopathol 1999;
11(3): 607-28.
9. Strauss AA, Lehtinen LE. Psychopathology and education of the brain-injured child. New York: Grune and Stratton;
1947.
10. Sideman LJ, Lieberman J, Milberg SH, Norman D, Seired K. Effect of family history and comorbidity on the
neuropsychological performance of children with ADHD. Am Acad Child Adolesc Psychiatry 1995; 34(8): 1015-24.
11. Woods SP, Lovejoy DW, Ball JD. Neuropsychological characteristics of adults with ADHD: a comprehensive
review of initial studies. Clin Neuropsychol 2002; 16: 12-34.
12. Boonstra AM, Oosterlaan J, Sergeant J, Buitelaar J. Executive functioning in adult ADHD: a meta-analytic review.
Psychol Med 2005; 35: 1097-108.
13. Dobson-Patterson R, O’Gorman J, Chan R, Shum D. ADHD subtypes and neuropsychological performance in an
adult sample. Res Dev Disabil 2016; 55: 55-63.
14. Pennington B. Toward a new neuropsychological model of attention deficit/hyperactivity disorder: Subtypes
and multiple deficits. Biol Psychiatry 2005; 7: 1221-3.
15. Perner J, Lang B, Kilo D. Theory of mind and self-control: More than a common problem of inhibition. Child Dev
2002; 73: 752-67.
16. Teeter PA, Wender EH. Attention-deficit hyperactivity disorder on adolescence. J Dev Behav 1998; 30: 5.
17. Pitcher TM, Piek JP, Hay DA. Fine and gross motor ability in males with ADHD. Dev Med Child Neurol 2003; 45:
525-35.
18. Laasonen M, Salomaa J, Cousineau D, Leppämäki S, Tani P, Hokkanen L, et al. Project DyAdd: Visual attention in
adult dyslexia and ADHD. Brain Cogn 2012; 80: 311-27.
19. Saeidi M, Bafandeh H, Noorazar GH. [Comparison of theory of mind in ADHD and control group. Proceeding of 5th
International Congress of Child Psychiatry. Tehran: Iran, 2012. (Persian)
THEORY OF MIND AND ADHD ABAZARI, MAHDAVI, DARVISHI
Fundamentals of Mental Health, 2017 Jan-Feb http://jfmh.mums.ac.ir 29
20. Hughes C, Ensor R. Executive function and theory of mind: Predictive relations from ages 2 to 4. Dev Psychol
2007; 43: 1447-59.
21. Gatica F, Redin C, Abaitua Ch. Emotional and cognitive profile of adolescents with ADHD: Effects of learning
mediated interaction. Procedia Soc Behav Sci 2013; 84: 1704-11.
22. Hughes C, Dunn J, White A. Trick or treat? Uneven understanding of mind and emotion and executive dysfunction
in hard-to-manage preschoolers. J Child Psychol Psychiatry 1998; 39: 981-94.
23. Grosbois N, Houssa N, Mazzone M. How could Theory of Mind contribute to the differentiation of social
adjustment profiles of children with externalizing behavior disorders and children with intellectual disabilities? Res Dev
Disabil 2013; 34: 2642-60.
24. Mize J, Pettit GS. Social information-processing and the development of conduct problems in children and
adolescents: Looking beneath the surface. In: Sharp C, Fonagy P, Goodyer I. (editors). Social cognition and
developmental psychopathology. New York: Oxford University Press; 2008: 141-74.
25. Pettit GS, Dodge KA, Brown MM. Early family experience, social problem solving patterns, and children’s social
competence. Child Dev 1998; 59: 107-20.
26. Buitelaar JK, Van Deer Wees M, Swaab-Barneveld H, Jan Van Deer Gaag R. Theory of mind and emotionrecognition functioning in autistic spectrum disorders and in psychiatric control and normal children. Dev Psychopathol
1999; 11: 39-58.
27. Hughes C, Leekam S. What are the links between theory of mind and social relations? Review and reflections and
new directions for studies of typical and atypical development. Soc Dev 2004; 13: 590-619.
28. Jolliffe D, Farrington DP. Examining the relationship between low empathy and bullying. Aggress Behav 2006; 32:
540-50.
29. Perugini EM, Harvey EA, Lovejoy DW, Sandstrom K, Webb AH. The predictive power of combined
neuropsychological measures for attention-deficit/hyperactivity disorder in children. Child Neuropsychol 2000; 6: 101-
14.
30. Ruff ME. Attention deficit disorder and stimulant use: An epidemic of modernity. Clin Pediatr (Phila) 2005; 44(7):
557-63.
31. Roth RW, Saykin AJ. Executive dysfunction in attention deficit/ hyperactivity disorder: Cognitive and neuromaging
finding. Psychiatr Clin North Am 2004; 27(1): 83-96.
32. Walker S. Gender differences in the relationship between young children's peer-related social competence and
individual differences in theory of mind. Genet Psychol 2005; 166(3): 297-312.
33. Muris P, Stresemann P, Messters C, Marckelbach H, Horselenberg R, Van Den-Hogen T, et al. The tom test: A new
instrument for assessing theory of mind in normal children with pervasive developmental disorders. J Autism Dev
Disord 1999; 29(1): 67-80.
34. Qmrany A, Alborzi Sh, Khayer M. [Study of validity and reliability of theory of mind test in a group of mental
retardation and normal students in Shiraz]. Journal of psychology 2006; 10(38): 181-99. (Persian)
35. Jadidi M, Abedi A. [Accommodation and normalization of Connor neuropsychology inventory in children with 5-12
years old in Isfahan]. Isfahan, Iran: University of Isfahan. (Persian)
36. Farid F, Kamkari K, Safarnya M, Afrooz S. [Comparison of validity of Tehran-new IQ Inventory, Stanford-Binet
inventory and fourth version of Wechsler intelligence scale among children with learning disabilities. Journal of
learning disability 2013; 4: 70-83. (Persian)
37. Shiri Aminloo M , Kamkary K, Shokrzadeh S. [The concurrent validity of the new version of the Tehran-StanfordBinet Intelligence Scale with the Wechsler Intelligence Scale for Children-Revised]. Exceptional education 2013; 7: 50-
61. (Persian)
38. Sprafkin J, Gadow KD, Salisbury H, Schneider J, Long J. Further evidence of reliability and validity of the child
symptom Inventory-4: parent checklist in clinically referred boys. Department of psychiatry, state university of New
York, NY; 2002: 11794-879.
39. Alipour A, Mohammad Esmail E. [Credit check and determine cut-off points disorder Child Symptom Inventoryyear-old CSI-4 on elementary and secondary school students 6-14 year in Tehran]. Tehran: Exceptional Children
Research Institute; 2003. (Persian)
40. Capage L, Watson AC. Individual differences in theory of mind, aggressive behavior, and social skills in young
children. Early Educ Dev 2001; 12: 613-28.
41. Marsh AA, Blair RJR. Deficits in facial affect recognition among antisocial populations: A meta-analysis. Neurosci
Biobehav Rev 2008; 32: 454-65.
42. Razavieh A, Latifian M, Arefi M. The role of theory of mind and empathy in predicting students overt and sociable
aggressive behavior. Journal of psychological study 2007; 2(3-4): 25-38.
Copyright of Journal of Fundamentals of Mental Health is the property of Mashhad
University of Medical Sciences, Center Office of Fundamentals of Mental Health and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission. However, users may print, download, or email
articles for individual use.
Introduction
T
HERE IS relatively high prevalence of
attentional difficulties amongst children
and young people in the UK, with Ford,
Goodman and Meltzer (2003) reporting that
3.62 per cent of boys and 0.85 per cent of
girls in the UK have Attention/Deficit Hyperactivity Disorder (ADHD), with many cases
persisting into adulthood (Lara et al., 2009).
Attentional difficulties, such as ADHD, are
also associated with peer rejection and low
self-esteem (Klassen, Miller & Fine, 2004), as
well as social anxiety and stress (Elkins et al.,
2011). Research suggests that impoverished
environmental input within the classroom
can prevent primary-aged pupils from realising their underlying attentional potential
(see Ruff & Rothbart, 1996; Schweizer, Moosbrugger & Goldhammer, 2005), which may
then compromise their cognitive development and academic progress (see Breslau et
al., 2010; Steele et al., 2012).
A neuropsychological model of attention
Attention ‘is the process by which certain information is selected for further processing and other
information is discarded’ (Ward, 2006, p.130),
and is characterised by its limited capacity
and responsivity to both sensory (auditory
and visual) and semantic stimulus characteristics (Lezak, Howieson & Loring, 2004,
p.34).
An early neuropsychological model of
attention by Luria (1973) postulated two
parallel attentional systems: the reflexive
attentional system (RAS) and the volitional
attentional system (VAS). The RAS responds
to biologically meaningful environmental
stimuli, whilst the VAS, characterised by individuals’ interpretations of environmental
stimuli, requires higher-order cognition to
work efficiently. It is widely accepted that
attentional skills improve with age (Cooley &
Morris, 1990), with Luria (1973) stating that
Educational & Child Psychology Vol. 33 No. 1 51
© The British Psychological Society, 2016
Measuring the effectiveness of a
mindfulness-based intervention for
children’s attentional functioning
George Thomas & Cathy Atkinson
Aim: This study sought to evaluate the impact of a six-hour mindfulness programme (Paws .b) on
mainstream primary school aged pupils’ suppressing and sustaining attention skills.
Method: A randomised control trial (RCT) design with quasi-experimental intervention cross-lag was used
with two classes of 8- to 9-year-olds. Pupils and class teachers were randomly assigned to the experimental
or waitlist control group, with each receiving the intervention over six weeks. Teacher-reported and
standardised attention measures were gathered at pre- and post-intervention, and at follow-up.
Findings: Within-condition comparisons revealed several significant pre- vs. post-intervention effects, the
majority of which were maintained at follow-up. Between-condition comparisons revealed some significant
partial condition × time-point interactions.
Limitations: Quasi-experimental research within ecologically valid real-world settings is prone to the
influence of confounding variables. In this case, a change of class teacher within the waitlist control group
may have had a marked impact on the findings.
Conclusions: The present study offers tentative evidence of the effectiveness of Paws .b in improving
attentional functioning in primary-aged children. Findings are discussed relative to mindfulness and
attention literature, and further implications for future research are outlined.
Keywords: mindfulness; attention; classroom intervention; cross-lag; quasi-experiment.
the RAS is present from early infancy, with
the VAS emerging later in childhood. Ruff
and Rothbart (1996) also identified an interactional process, whereby neural substrates
dictate potential attentional development
throughout infancy and childhood, with environmental input determining the extent to
which underlying potential is realised. Therefore, the classroom potentially serves as the
ideal environment in which to develop
pupils’ attention, particularly as ‘schoolchildren and adolescents of all levels of intelligence
have difficulties in attention’ (Das, 2002, p.241).
Strauss et al. (2000) stated that a
neuropsychological approach to attention is
characterised by separating the parameters
of attention into independent components,
relative to performance on various neuropsychological tasks. There are five attentional
components: suppressing attention; sustaining attention; focusing attention; shifting
attention; and divided attention (see
Anderson et al., 2002; Mirsky et al., 1991;
Stuss et al., 1995). Working definitions of
each are shown in Table 1 below.
Thomas (2013) asked Special Educational Needs Co-ordinators (SENCOs) about
which attentional components were most
commonly used and required by mainstream
primary-aged pupils. Findings indicated that
SENCOs viewed sustaining attention (Mirsky
et al., 1991) and supressing attention (Stuss
et al., 1995) as most important. This
provided a rationale for the present study’s
focus on suppressing and sustaining attention skills within the classroom. Additionally,
MacKay (2005) offers a rationale for the use
of a neuropsychological approach within an
educational setting, in that ‘neuropsychology
should not be seen as a specialist “bolt-on” area
within educational psychology, but rather as part
of the essential knowledge and practice base
required for the effective delivery of EP services’
(p.7).
Mindfulness-based attention state training
Petersen and Posner (2012) suggest there
are two broad methods of attentional
improvement: attention training (practice
using a particular attentional network) and
attention state training (practice using forms
of meditation to alter the neural state in a
way that improves attention). Tang and
Posner (2009) highlight that the focus of
attention training is to use cognitive tasks to
practise using particular attentional
networks, whereas the focus of attention
state training is for experiences, such as
mindfulness, to effect the mind and improve
performance. Given the prominence of
experiential learning within the primary
National Curriculum (Department for
Education [DfE], 2013), it was felt that attention state training would be a more appropriate focus for the present study.
Mindfulness has its origins in many
cultural, contemplative, and philosophical
traditions, particularly Buddhism (KabatZinn, 2000), and is defined as ‘the awareness
that emerges through paying attention on purpose,
52 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
Table 1: Working definitions of the five attentional components.
Attentional component Definition
Suppressing attention The ability to inhibit prepotent responses, or to supress impulsive
responses
Sustaining attention The capacity to maintain focus and alertness over time, or vigilance
Focusing attention The ability to select target information from an array for enhanced
processing
Shifting attention The ability to change attentive focus in a flexible and adaptive manner
Divided attention When two or more tasks have to be carried out at the same time
in the present moment and non-judgementally’
(Kabat-Zinn, 2003, p.45). Typically, mindfulness practices include breathing, body scan,
movement, and listening exercises whilst
ensuring that one supresses their attention
towards intrusive thoughts/judgements/
mind wanderings (Williams & Penman,
2011). Whilst the Melbourne Academic
Mindfulness Interest Group (MAMIG, 2006)
proposes that mindfulness is a metacognitive
skill, others assert that it also involves
suppressing and sustaining attention (see
Bishop et al., 2004; Lutz et al., 2008).
Jones (2011) suggests mindfulness has as
much of a place within the education system
as exams, as it prepares pupils for life outside
of school. Becuase of its secular nature, mindfulness should not clash with pupils’ religious
or cultural beliefs and thus should be implementable within a multicultural classroom
(Lubelska, 2012). Goodman (2005) and
Kabat-Zinn (1990) suggest that children’s
openness to experience, readiness to learn,
and creativity make them more likely to
benefit from mindfulness than adults.
There are a growing number of studies
citing the positive effects of mindfulness on
children’s and adolescents’ attentional functioning (Felver et al., 2014; Semple et al.,
2010; van der Oord, Bögels & Peijnenburg,
2012). However, Greenberg and Harris
(2012) noted that few trials with children
have been conducted with mainstream
populations and that those which exist have
inherent problems with design, sample size
and measurement.
One exception is a study by Napoli,
Krech and Holley (2005) who evaluated the
impact of the Attention Academy
Programme, a manualised mindfulness
programme delivered by experienced mindfulness practitioners. The RCT design
involved 228 6- to 9-year-old mainstream
pupils randomly allocated to either an
experimental group or a control group.
Teachers completed pre- and post-intervention measures of pupils’ ADHD symptoms
using the ADD-H Comprehensive Teacher
Rating Scale (ACTeRS; Ullmann, Sleator &
Sprague, 1997), whilst pupils undertook
specific subtests from the Test of Everyday
Attention for Children (TEA-Ch; Manly et
al., 2001) pertaining to focusing and
sustaining attention. Findings demonstrated
that the experimental group had significantly lower teacher-rated ADHD symptoms
(r=0.49) and better focusing attention
performance (r=0.6) at post-intervention
compared to controls, although no significant difference was observed between both
groups at post-intervention in terms of
sustaining attention. However, limitations
included: the use of the ACTeRS, which is
tailored towards the identification of ADHD
symptoms and is not intended for use with
typically-developing children; the lack of
baseline between-condition comparisons;
and, the lack of follow-up assessment.
Mindfulness in schools
A debate exists as to whether mindfulness
teachers need to be established mindfulness
practitioners in order to teach mindfulness
effectively to children. Kostanski and Hassed
(2008) highlight this as a tradition but note
that robust scientific evidence for this is
limited. The MAMIG (2006) suggests that
mindfulness is traditionally taught in groups,
but with individual support, as this is costeffective; allows pupils to learn from one
another; and, allows greater pupil motivation due to peer support.
In terms of the intervention used in this
study, a well-established nine-week manualised mindfulness programme for teenagers
called ‘.b for teens’ (Mindfulness in Schools
Project, 2013b) was already being delivered
locally in secondary schools and sixth form
colleges. Following its success (Huppert &
Johnson, 2010; Kuyken et al., 2013), the
Mindfulness in Schools Project differentiated the .b for teens materials so that
primary-aged pupils could access the
curriculum content and mindfulness practices. This formed the basis of a new six-week
programme called ‘Paws .b’ (Mindfulness in
Schools Project, 2013a). The aim of the
present study was to evaluate the impact of
Educational & Child Psychology Vol. 33 No. 1 53
Measuring the effectiveness of a mindfulness-based intervention for children’s attentional functioning
the Paws .b mindfulness programme on
mainstream primary-aged pupils’ suppressing and sustaining attention skills within the
UK context, and, in doing so, address the
following research question:
To what extent does the Paws .b mindfulness
programme lead to an improvement in mainstream primary school aged pupils’ suppressing
and sustaining attention skills?
Method
Sample
Convenience sampling was undertaken at a
mixed and ethnically diverse comprehensive
primary school in north-west England where
the first researcher was the link Educational
Psychologist (EP). Prior to the beginning of
the present study, the school had independently decided to introduce Paws .b.
The researchers made an ethical decision
to target Year 4 pupils (8- and 9-year-olds)
who are unlikely to be involved in preparation for Standard Assessment Tests (SATs)
taken in Year 6. Thirty Year 4 pupils were
recruited (mean age 8 years 10 months; 50
per cent female): 16 from class one (mean
age 8 years 9 months; 44 per cent females)
and 14 from class two (mean age 8 years
10 months; 57 per cent female). Both class
teachers were female. However, it should be
noted that the original waitlist control
teacher went on maternity leave half way
through the study and was replaced by
another female class teacher. The mindfulness teacher who delivered Paws .b to both
groups was an established mindfulness practitioner and had received training to deliver
Paws .b from the Mindfulness in Schools
Project.
Relative to the geographical location of
the school, all pupils had English as an additional language (EAL) with a predominance
of Urdu or Bengali as their first language.
However, through consultation with class
teachers and the mindfulness teacher, the
language demands of Paws .b were deemed
to be in line with the Year 4 National
Curriculum (DfE, 2013).
Design and procedure
A RCT design with a quasi-experimental
intervention cross-lag was utilised (see
Semple et al., 2010) with four data collection
time-points six to eight weeks apart: Baseline; Time 1; Time 2; and Time 3. Prior to
Baseline, the coin toss method was used to
randomly allocate class one to the experimental group, leaving class two in the waitlist
control group. The experimental group
received Paws .b between Baseline and Time1; the waitlist control group received Paws .b
between Time 1 and Time 2. Hence, the
intervention cross-lag took place at Time 1
when the mindfulness teacher cross-lagged
from delivering Paws .b to the experimental
group to delivering Paws .b to the waitlist
control group.
Within the experimental group, Time 2
and Time 3 acted as the eight-week and
14-week post-intervention follow-ups. For the
waitlist control group, Time 3 acted as the
six-week post-intervention follow-up. The
different follow-up periods reflected the
length of each school half-term. The relationship between the four time-points and
the intervention stages per condition are
represented in Table 2.
Materials
Paws .b consists of six one-hour lessons that
are delivered on a weekly basis. The materials consist of a Microsoft PowerPoint presentation and plan for each lesson. An
overview of Paws .b is represented in Table 3.
Implementation checks were carried out by
the first researcher throughout the two intervention periods (see Barnett, Hawkins &
Lentz Jr., 2011; Durlak & DuPre, 2008).
Measures and scoring
The teacher-reported attention measure
used within the present study was the
Attention Checklist (Das, 2002), which the
class teachers completed for all pupils at
each of the four time-points. It is a 12-item
questionnaire that pertains to observable
attention behaviours that pupils are likely to
exhibit within the classroom (e.g. ‘Does the
54 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
child daydream in class?’). Each attention
behaviours is rated using four descriptors:
‘Not at all’; ‘Just a little’; ‘Pretty much’; and,
‘Very much’; and is coded so that positive
attention behaviours receive higher scores.
The Attention Checklist was chosen due to it
being high in face validity relative to
assessing pupils’ suppressing and sustaining
attention skills.
The change of class teacher within the
waitlist control group took place immediately after Time 1. In an attempt to protect
the validity of the Attention Checklist
dataset, the second waitlist control teacher
was provided with the Attention Checklist
data which had been generated by the
original waitlist control teacher at Baseline
and Time 1. Through cross-standardisation,
the researchers hoped that the second waitlist control teacher would be able to establish
the range of attention behaviours typically
displayed by each pupil at Baseline and
Time 1 before carrying out their own ratings
at Time 2 and Time 3. Despite these efforts,
it is acknowledged that the change of class
teacher within the waitlist control group is a
confounding variable in the study.
The researcher-administered attention
measure used within the present study was
the NEPSY-II (Korkman, Kirk & Kemp,
2007) Inhibition subtest, which the first
researcher administered to all pupils at each
Educational & Child Psychology Vol. 33 No. 1 55
Measuring the effectiveness of a mindfulness-based intervention for children’s attentional functioning
Table 2: Relationship between time-points and intervention stages per condition.
Condition Intervention stage
Baseline Time 1 Time 2 Time 3
Experimental group Pre- Post- Follow-up 1 Follow-up 2 intervention intervention
Waitlist control group Pre pre- Pre- Post- Follow-up 1 intervention intervention intervention
Table 3: Overview of the Paws .b mindfulness programme.
Lesson number Description of content
1 l An introduction to the brain;
l A discussion of our ability to make choices;
l A breath counting mindfulness exercise.
2 l An introduction to the ‘searchlight’ of attention;
l An introduction to the philosophy of mindfulness;
l Two mindful breathing exercises.
3 l A grounding mindfulness exercise;
l A discussion of ‘wobbly feelings’.
4 l The ‘count and add two’ mindful breathing exercise;
l A discussion of how to avoid reacting badly to situations.
5 l A discussion of worries;
l A discussion of how worries can be supported by the previously learnt
mindfulness exercises.
6 l A recap of learning;
l Practise all learnt mindfulness exercises.
of the four time-points. As the NEPSY-II Inhibition subtest includes several tasks, the
researchers selected the Naming and Inhibition Total Errors tasks due to them being
high in face validity relative to assessing
pupils sustaining and suppressing attention
skills, respectively (see Table 4). Although
the researchers considered using the TEACh (Manly et al., 2001) as in Napoli et al.’s
(2005) study, the NEPSY-II Inhibition subtest
appeared more sensitive to pupils’
suppressing and sustaining attention skills
and thus had superior face validity when
compared to the TEA-Ch.
The total number of errors made within
each of the two tasks are recorded and used
to generate one of seven percentile rank
categories relative to pupils’ age group: <2;
2–5; 6–10; 11–25; 26–50; 51–75; and >75.
The percentile rank categories were coded
so that pupils who made fewer errors
received higher scores. Hence, the lowest
percentile rank category (<2) received a
score of 1 and the highest percentile rank
category (>75) received a score of 7. Pre preintervention vs. pre-intervention withincondition comparisons were carried out in
the waitlist control group in order to rule out
the presence of practice effects and ensure
the validity of the measures.
In summary, three variables were created:
Attention Checklist; Naming Total Errors;
and, Inhibition Total Errors.
Findings
Teacher-reported Attention Checklist measure:
within-condition comparisons
Within the experimental group, Attention
Checklist scores increased significantly from
pre-intervention (median=42.5) to post-intervention (median=45). Whilst this intervention
effect was not maintained at the eight-week
follow-up (median=41.5), it was re-maintained
at the 14-week follow-up (median=46) with a
large estimated effect size1. Within the waitlist
control group, Attention Checklist scores did
not increase significantly from pre pre-intervention (median=38.5) to pre-intervention
(median=36.5), yet decreased significantly
from pre-intervention to post-intervention
(median=33.5). Whilst this inverse intervention effect was an unexpected finding, it was
not maintained at the six-week follow-up
(median=38). Findings achieving statistical
56 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
Table 4: Description of the Naming and the Inhibition Total Errors tasks.
Naming Total Errors task Inhibition Total Errors task
Shapes picture matrix Name each shape (i.e. ‘circle’/ Same procedure as the Naming
containing circles ‘square’) looking across each row Total Errors task, except pupils
and squares from left to right, starting with had to say ‘square’ when they saw
the top row and finishing with a circle and ‘circle’ when they saw
the bottom row. a square.
Arrows picture matrix State the direction of each arrow Same procedure as the Naming
containing arrows (i.e. ‘up’/ ‘down’) looking across Total Errors task, except pupils had
pointing up or down each row from left to right, to say ‘down’ then they saw an
starting with the top row and up arrow and ‘up’ when they saw
finishing with the bottom row. a down arrow.
1 An effect size is a standardised measure of the magnitude of an observed effect (Field, 2005, p.32). In the
present study, Rosenthal’s (1991, p.19) method was used to calculate estimated effect sizes using the formula
r=Z÷(√n). Hence, effect sizes are denoted by ‘r’. Given the present sample size was ≥28 but <85, Cohen (1992)
states that there was only sufficient statistical power to detect large estimated effect sizes (i.e. r≥.5).
significance on the Wilcoxon signed-rank test
(Wilcoxon, 1945) are indicated with an
asterisk (see Table 5).
Teacher-reported Attention Checklist measure:
between-condition comparisons
At Baseline, Attention Checklist scores within
the experimental group (median=42.5) did
not differ significantly from the waitlist
control group (median=38.5). At Time 1,
scores within the experimental group
(median=45) were significantly higher than
the waitlist control group (median=36.5), as
might have been predicted. However, unexpectedly the between-condition difference
remained at Time 2 (experimental group
median=41.5, waitlist control group
median=33.5) after the waitlist control group
had received Paws .b, and also remained at
Time 3 (experimental group median=46,
waitlist control group median=38). Findings
achieving statistical significance on the
Mann-Whitney test (Mann & Whitney, 1947)
are indicated with an asterisk (see Table 6).
Researcher-administered NEPSY-II measures:
within-condition comparisons
Within the experimental group, Naming
Total Errors scores increased significantly
from pre-intervention (median=4) to postintervention (median=6), meaning they
made fewer errors at post-intervention.
Furthermore, this intervention effect was
maintained at the eight-week follow-up
(median=6) and 14-week follow-up
(median=7) with large estimated effect sizes.
Within the waitlist control group, Naming
Total Errors scores did not increase
significantly from pre pre-intervention
(median=3.5) to pre-intervention
(median=4.5), yet increased significantly
from pre-intervention to post-intervention
(median=7). Nonetheless, this intervention
effect was not maintained at the six-week
follow-up (median=5.5).
Within the experimental group, Inhibition Total Errors scores increased significantly from pre-intervention (median=2.5)
to post-intervention (median=5). This interEducational & Child Psychology Vol. 33 No. 1 57
Measuring the effectiveness of a mindfulness-based intervention for children’s attentional functioning
Table 5: Teacher-reported measure inferential statistics: within-condition comparisons.
Variable Condition Pre pre- vs. pre- Pre- vs. post- Maintained at Maintained at
intervention intervention at follow-up 1 follow-up 2
Attention Experimental N/A p=.012* n.s. p=.001**
Checklist group r=–.53
Waitlist n.s. p=.039* n.s. N/A
control group (inverse)
Note: n.s.=not significant; *=p<.05; **=p<.01; N/A=impossible comparison due to the nature of the intervention cross-lag.
Table 6: Teacher-reported measure inferential statistics:
between-condition comparisons.
Variable Predicted outcome
No significant Significant difference at No significant No significant
difference at Time 1: Experimental difference at difference at
Baseline group > waitlist Time 2 Time 3
control group
Attention
n.s. p=.033* p=.039* p=.026* Checklist
Note: n.s.=not significant; *=p<.05; predicted outcomes have no shading; unpredicted outcomes are shaded grey.
vention effect was maintained at the eightweek follow-up (median=6) and 14-week
follow-up (median=5) with a large estimated
effect size at 14-week follow-up. Within the
waitlist control group, Inhibition Total
Errors scores did not increase significantly
from pre pre-intervention (median=4)
to pre-intervention (median=2.5), yet
increased significantly from pre-intervention
to post-intervention (median=6.5). Nonetheless, this intervention effect was not maintained at the six-week follow-up (median=6).
Findings achieving statistical significance on
the Wilcoxon signed-rank test are indicated
with an asterisk (see Table 7)
Researcher-administered NEPSY-II measures:
between-condition comparisons
At Baseline, Naming Total Errors scores
within the experimental group (median=4)
did not differ significantly from scores within
the waitlist control group (median=3.5). At
Time 1, scores within the experimental
group (median=6) were significantly higher
than scores within the waitlist control group
(median=4.5) and the between-condition
difference disappeared at Time 2 (experimental group median=6, waitlist control
group median=7), as might have been
predicted. However, the between-condition
difference unexpectedly reappeared at Time
3 (experimental group median=7, waitlist
control group median=5.5) (see Table 8)
At Baseline, Inhibition Total Errors
scores within the experimental group
(median=2.5) did not differ significantly
from scores within the waitlist control group
(median=4). However, unexpectedly the lack
of a between-condition difference remained
at Time 1 (experimental group median=5,
waitlist control group median=2.5), Time 2
(experimental group median=6, waitlist
control group median=6.5), and Time 3
(experimental group median=5, waitlist
control group median=6). Findings
achieving statistical significance on the
Mann-Whitney test are indicated with an
asterisk (see Table 8).
Discussion
There are several findings which provide
tentative evidence as to the positive impact
of Paws .b on mainstream primary-aged
pupils’ attentional functioning. Measures
taken from the Attention Checklist and the
Naming and Inhibition Total Errors tasks
indicated that Paws .b had a significantly
positive immediate and sustained impact
upon the attentional functioning of pupils in
the experimental group. According to the
Attention Checklist, the experimental
teacher observed pupils exhibiting more
58 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
Table 7: Researcher-administered measures inferential statistics:
within-condition comparisons
Variable Condition Pre pre- vs. pre- Pre- vs. post- Maintained at Maintained at
intervention intervention at follow-up 1 follow-up 2
Naming Experimental N/A p=.003** p=.002** p<.001**
Total group r=–.56 r=–.56
Errors Waitlist n.s. p=.022** n.s. N/A
control group
Inhibition Experimental N/A p=.015* p=.003** p=.001**
Total group r=–.52
Errors Waitlist n.s. p=.01* n.s. N/A
control group
Note: n.s.=not significant; *=p<.05; **=p<.01; N/A=impossible comparison due to the nature of the intervention cross-lag.
desirable suppressing and sustaining attention behaviours at post-intervention and
14-week follow-up when compared to preintervention. Likewise, according to Naming
and Inhibition Total Errors tasks, the experimental pupils were more accurate in their
use of sustaining and suppressing attention
skills at post-intervention and follow-up
when compared to pre-intervention. The
Naming and Inhibition Total Errors tasks
also indicated that Paws .b had a significantly
positive immediate impact upon the attentional functioning of pupils in the waitlist
control group and suggested that they were
more accurate in their use of sustaining and
suppressing attention skills at post-intervention when compared to pre-intervention.
Within the findings were two clear
patterns which are worthy of further discussion, both of which were unexpected
outcomes:
1. Intervention effects were stronger in the
experimental group than the waitlist
control group;
2. Follow-up effects within the experimental
group were stronger than immediate
intervention effects.
With regards to intervention effects being
stronger in the experimental group than the
waitlist control group, there are three
possible explanations. Firstly, the original
waitlist control teacher went on maternity
leave immediately after Time 1, meaning
that waitlist control pupils received Paws .b at
the same time as having to adjust to, and
build new relationships with, the second
waitlist control teacher. This disruption is
likely to have reduced the impact of Paws .b
within the waitlist control group.
Secondly, experimental and waitlist
control pupils were not matched according
to gender or core personality traits (e.g.
agreeableness/conscientiousness/openness
to experience; Norman, 1963). It is possible
that the gender differences between the
groups may have confounded the impact of
Paws .b, although research has yet to establish whether gender affects the impact of
mindfulness programmes, and given the
small sample sizes within each group, it is
questionable what impact gender differences may have had on the findings. Furthermore, had personality measures been taken,
these may have revealed differences between
the groups which may have affected the
impact of Paws .b.
Thirdly, pupils’ dispositional mindfulness
(the extent to which pupils naturally pay
attention on purpose, in the present
moment and non-judgementally) was not
assessed. Hence, the researchers were
unable to carry out mediation analyses and
Educational & Child Psychology Vol. 33 No. 1 59
Measuring the effectiveness of a mindfulness-based intervention for children’s attentional functioning
Table 8: Researcher-administered measures inferential statistics:
between-condition comparisons.
Variable Predicted outcome
No significant Significant difference at No significant No significant
difference at Time 1: Experimental difference at difference at
Baseline group > waitlist Time 2 Time 3
control group
Naming
Total n.s. p=.009** n.s. p=.008**
Errors
Inhibition
Total n.s. n.s. n.s. n.s.
Errors
Note: n.s.=not significant; *=p<.05; predicted outcomes have no shading; unpredicted outcomes are shaded grey.
ascertain whether pupils’ dispositional mindfulness mediated the relationship between
Paws .b and their suppressing and sustaining
attention skills. Therefore, intervention
effects within the experimental group may
have been stronger due to them having
greater dispositional mindfulness at preintervention.
With regards to follow-up effects within
the experimental group being stronger than
immediate intervention effects, there is one
possible explanation. With reference to
Haring and Eaton’s (1978) instructional
hierarchy of skill development, experimental pupils were able to acquire, become
fluent in their use of, and maintain their use
of, mindfulness skills throughout Paws .b.
Between post-intervention and follow-up,
experimental pupils would have had additional opportunities to generalise mindfulness skills in novel situations and adapt them
to suit their everyday lives. This may explain
why large estimated effect sizes were
observed at follow-up and not post-intervention.
Relative to previous research, the present
study replicated the findings of Napoli et al.
(2005). Both studies used a RCT design to
evaluate the impact of a manualised mindfulness programme on mainstream primaryaged pupils’ attentional functioning. Using
between-condition comparisons, both
studies found that experimental pupils had
significantly higher post-intervention attention scores than controls on teacherreported and researcher-administered
measures of attention. The present study
also made a number of methodological
improvements relative to Napoli et al.
(2005). Specifically, the present study: used a
teacher-reported attention measure tailored
towards observable attention behaviours (i.e.
the Attention Checklist) as opposed to
ADHD symptoms (i.e. the ACTeRS);
included baseline between-condition
comparisons to ensure that the experimental
and control groups did not differ significantly at the outset; and, included follow-up
assessment.
However, it should be noted that, despite
attempts to introduce methodological rigour,
the present study illustrates some of the
issues facing EPs as real-world researchers in
trying to implement quasi-experimental
designs. From a positive perspective, the use
of a RCT with a quasi-experimental intervention cross-lag enabled the researchers to
establish a link between the introduction of
Paws .b and associated changes in pupils’
attentional functioning, as well as reduce
possible Hawthorne effects by ensuring that
that Paws .b was delivered to all pupils by the
same mindfulness teacher. However, the use
of a RCT in a real-world setting was not only
difficult to negotiate with school stakeholders, but was difficult to implement in
practice. As there was a lack of a double-blind
element to the design, class teachers and the
first researcher knew when each group of
pupils were receiving Paws .b, which may, in
turn, have increased possible Hawthorne
effects by affecting their ratings/ assessments
of pupils attentional functioning. Furthermore, the present study failed to achieve
Greenberg and Harris’ (2012) criterion for a
long-term follow-up of six months.
Upon critiquing the sample used within
the present study, the researchers were
restricted to inferring only large estimated
effect sizes because of the small number of
pupils and the experimental and waitlist
control groups were unmatched according
to gender and core personality traits. The
generalisability of the findings were low due
to the sample consisting entirely of pupils
who had EAL and due to the study taking
place in a single context. Furthermore, there
was unavoidable attrition of the original waitlist control group teacher. Although steps
were taken to protect the validity of the
Attention Checklist dataset within the waitlist control group, the validity of comparing
the Attention Checklist data generated by
the original waitlist control teacher against
the data generated by the second waitlist
control teacher is questionable and may
explain why an inverse intervention effect
was observed.
60 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
It is the researchers’ view that the single
largest difficulty within the present study was
controlling for extraneous variables, relative
to its quasi-experimental design. For
example, it is very likely that the factor which
most affected the impact of Paws .b on the
waitlist control group (and subsequently the
RCT design as whole) was their change of
class teacher. Campbell and Stanley (1963)
state that quasi-experimental designs allow
researchers to control the ‘who and to whom of
measurement’, but lack control over the ‘when
and to whom of exposure’, where randomly
assigning pupils to experimental and control
groups is an essential component of an RCT
design. However, in a real-world setting such
as a school, randomisation is often unacceptable and, therefore, it is difficult to exclude
the possibility of confounding variables
(Robitaille et al., 2012).
The findings may be useful to EPs in light
of the growing interest in mindfulness-based
interventions (Davis, 2012). The study
provides tentative evidence that, for schools,
Paws .b offers a means of improving the
attentional functioning of their pupils. Paws
.b also has the potential to provide additional support for individual and groups of
pupils with attentional difficulties such as
ADHD. At the whole-class level, Paws .b may
offer a means of enriching environmental
input within the classroom in a way that
enables pupils to realise their underlying
attentional potential (see Ruff & Rothbart,
1986; Schweizer et al., 2005), which may
promote their cognitive development and
academic progress (see Breslau et al., 2010;
Steele et al., 2012). Regarding educational
psychology practice, the researchers believe
that EPs are well placed to support schools’
implementation of preventative interventions such as Paws .b rather than necessarily
delivering mindfulness interventions directly
(see Fallon, Woods & Rooney, 2010).
There are numerous possible directions
for future research suggested by this paper.
It might be possible to replicate the present
study, and extend its findings by making
appropriate methodological adjustments
(for instance, by including a double-blind
element and a six-month follow-up within
the design; recruiting a larger sample of
pupils who are able to be matched according
to gender and core personality traits; or
recruiting a larger sample of class teachers to
help reduce the potential impact of attrition), or by replicating it with a sample of
non-EAL pupils in a different context. Other
directions include assessing whether pupils’
dispositional mindfulness mediates the
impact of Paws .b, or assessing the impact of
Paws .b on the attentional functioning of a
clinical group of pupils, such as those with
ADHD. It might also be interesting to
compare the impact of Paws .b against more
and less intensive mindfulness programmes,
as well as other forms of attention state
training, regarding pupils’ attentional functioning, or to observe the impact of Paws .b
at the neural level (using Functional
Magnetic Resonance Imaging, for example).
Address for correspondence
Cathy Atkinson
Doctorate in Educational and Child
Psychology,
Floor 5, Ellen Wilkinson Building,
The University of Manchester,
Manchester M13 9PL.
Email: Cathy.Atkinson@manchester.ac.uk
Educational & Child Psychology Vol. 33 No. 1 61
Measuring the effectiveness of a mindfulness-based intervention for children’s attentional functioning
Anderson, V., Northam, E., Hendy, J. & Wrennall, J.
(2002). Developmental neuropsychology: A clinical
approach. Hove: Psychology Press.
Barnett, D., Hawkins, R. & Lentz Jr., F.E. (2011).
Intervention adherence for researcher and practice: Necessity or triage outcome? Journal of
Educational and Psychological Consultation, 21(3),
175–190.
Bishop, S.R., Lau, M., Shapiro, S., Carlson, L.,
Anderson, N.D., Carmody, J., Segal, Z.V., Abbey,
S., Speca, M., Velting, D. & Devins, G. (2004).
Mindfulness: A proposed operational definition.
Clinical Psychology: Science and Practice, 11,
230–241.
Breslau, N., Breslau, J., Peterson, E., Miller, E., Lucia,
V.C., Bohnert, K. & Nigg, J. (2010). Change in
teachers’ ratings of attention problems and
subsequent change in academic achievement:
A prospective analysis. Psychological Medicine, 40,
159–166.
Campbell, D.T. & Stanley, J. (1963). Experimental and
quasi-experimental designs for research on teaching.
Boston, MA: Houghton Mifflin.
Cohen, J. (1992). A power primer. Psychological
Bulletin, 112(1), 155–159.
Cooley, E.L. & Morris, R.D. (1990). Attention in
children: A neuropsychologically-based model
for assessment. Developmental Neuropsychology, 6,
239–274.
Das, J.P. (2002). Attention: Concept, tests and
teacher ratings. Psychology and Developing Societies,
14(2), 241–260.
Davis, T.S. (2012). Mindfulness-based approaches
and their potential for educational psychology
practice. Educational Psychology in Practice, 28(1),
31–46.
Department for Education (DfE) (2013). The school
curriculum. Retrieved 22 November 2013, from:
http://www.education.gov.uk/schools/
teachingandlearning/curriculum.
Durlak, J.A. & DuPre, E.P. (2008). Implementation
matters: A review of research on the influence of
implementation on program outcomes and the
factors affecting implementation. American
Journal of Community Psychology, 41, 327–350.
Elkins, I.J., Malone, S., Keyes, M., Iacono, W.G. &
McGue, M. (2011). The impact of attentiondeficit/hyperactivity disorder on preadolescent
adjustment may be greater for girls than boys.
Journal of Clinical Child and Adolescent Psychology,
40(4), 532–545.
Fallon, K., Woods, K. & Rooney, S. (2010). A discussion of the developing role of educational
psychologists within Children’s Services. Educational Psychology in Practice, 26(1), 1–23.
Felver, J.C., Tipsord, J.M., Morris, M.J., Racer, K.H. &
Dishion, T.J. (2014). The effects of mindfulnessbased intervention on children’s attention regulation. Journal of Attention Disorders, 1–10.
Field, A. (2005). Discovering statistics using SPSS
(2nd ed.). London: Sage.
Ford, T., Goodman, R. & Meltzer, H. (2003). The
British child and adolescent mental health survey
1999: The prevalence of DSM-IV disorders.
Journal of the American Academy of Child and Adolescent Psychiatry, 42, 1203–1211.
Goodman, T.A. (2005). Working with children:
Beginner’s mind. In C.K. Germer, R.D. Siegel &
P.R. Fulton (Eds.), Mindfulness and psychotherapy.
New York: The Guilford Press.
Greenberg, M.T. & Harris, A.R. (2012). Nurturing
mindfulness and children and youth: Current
state of research. Child Development Perspectives,
6(2), 161–166.
Haring, N.G. & Eaton, M.D. (1978). Systematic
instructional procedures: An instructional hierarchy. In N.G. Haring, T.C. Lovitt, M.D. Eaton &
C.L. Hansen (Eds.), The Fourth R: Research in the
classroom. Ohio: Charles E. Merrill.
Huppert, F.A. & Johnson, D.M. (2010). A controlled
trial of mindfulness training in schools: The
importance of practice for an impact on wellbeing. The Journal of Positive Psychology, 5(4),
264–274.
Jones, D. (2011). Mindfulness in schools. The Psychologist, 24(10), 736–739.
Kabat-Zinn, J. (1990). Full catastrophe living: Using the
wisdom of your body and mind to face stress, pain, and
illness. New York: Delacorte.
Kabat-Zinn, J. (2000). Indra’s net at work: The mainstreaming of Dharma practice in society. York Beach,
ME: Samuel Weiser.
Kabat-Zinn, J. (2003). Mindfulness-based interventions in context: Past, present and future. Clinical
Psychology: Science and Practice, 10, 44–56.
Klassen, A.F., Miller, A. & Fine, S. (2004). Healthrelated quality of life in children and adolescents
who have diagnoses of attention-deficit/hyperactivity disorder. Pediatrics, 114(5), 541–547.
Korkman, M., Kirk, U. & Kemp, S. (2007). NEPSY–II.
Pearson Assessment.
Kostanski, M. & Hassed, C. (2008). Mindfulness as a
concept and a process. Australian Psychologist,
43(1), 15–21.
Kuyken, W., Weare, K., Ukoumunne, O.C., Vicary, R.,
Motton, N., Burnett, R., Cullen, C., Hennelly, S.
& Huppert, F. (2013). Effectiveness of the Mindfulness in Schools Programme: Non-randomised
controlled feasibility study. British Journal of
Psychiatry, 203(2), 126–131.
62 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
References
Lara, C., Fayyad, J., de Graaf, R., Kessler, R.C.,
Aguilar-Gaxiola, S., Angermeyer, M., Demytteneare, K., de Girolamo, G., Haro, J.M., Jin, R.,
Karam, E.G., Lepine, J.P., Mora, M.E., Ormel, J.,
Posada-Villa, J. & Sampson, N. (2009). Childhood predictors of adult attention-deficit/hyperactivity disorder: Results from the World Health
Organisation world mental health survey initiative. Biological Psychiatry, 65, 46–54.
Lezak, M.D., Howieson, D.B. & Loring, D.W. (2004).
Neuropsychological assessment. Oxford: Oxford
University Press.
Lubelska, A. (2012). Project report: Peaceful schools.
International Journal of Children’s Spirituality,
17(2), 187–191.
Luria, A.R. (1973). The working brain. New York: Basic
Books.
Lutz, A., Slagter, H.A., Dunne, J. & Davidson, R.J.
(2008). Attention regulation and monitoring in
meditation. Trends in Cognitive Sciences, 12(4),
163–169.
MacKay, T. (2005). The relationship of educational
psychology and clinical neuropsychology. Educational and Child Psychology, 22(2), 7–17.
Manly, T., Nimmo-Smith, I., Watson, P., Anderson, V.,
Turner, A. & Robertson, I.H. (2001). The differential assessment of children’s attention: The
Test of Everyday Attention for Children (TEACh): Normative sample and ADHD performance. Journal of Child Psychology and Psychiatry, 42,
1065–1081.
Mann, H.B. & Whitney, D.R. (1947). On a test of
whether one or two random variables is stochastically larger than the other. Annals of Mathematical
Statistics, 18, 50–60.
Melbourne Academic Mindfulness Interest Group
(MAMIG) (2006). Mindfulness-based psychotherapies: A review of conceptual foundations,
empirical evidence and practical considerations.
Australian and New Zealand Journal of Psychiatry,
40, 285–294.
Mindfulness in Schools Project (2013a). Paws .b.
Retrieved 22 November 2013, from:
http://mindfulnessinschools.org/
courses/paws-b/.
Mindfulness in Schools Project (2013b). .b for Teens.
Retrieved 22 November 2013, from:
http://mindfulnessinschools.org/
courses/b-for-teens/.
Mirsky, A.F., Anthony, B.J., Duncan, C.C., Ahearn,
M.B. & Kellam, S.G. (1991). Analysis of the
elements of attention: A neuropsychological
approach. Neuropsychology Review, 2(2), 109–145.
Napoli, M., Krech, P.R. & Holley, L.C. (2005). Mindfulness training for elementary school students:
The attention academy. Journal of Applied School
Psychology, 21(1), 99–125.
Norman, W.T. (1963). Toward an adequate taxonomy
of personality attributes: Replicated factor structure in peer nomination personality ratings.
Journal of Abnormal and Social Psychology, 66,
574–583.
Petersen, S.E. & Posner, M.I., (2012). The attention
system of the human brain: 20 years after. Annual
Review of Neuroscience, 35, 73–89.
Robitaille, Y., Fournier, M., Laforest, S., Gauvin, L.,
Filiatrault, J. & Corriveau, H. (2012). Effect of a
fall prevention programme on balance maintenance using a quasi-experimental design in realworld setting. Journal of Ageing and Health, 24,
827–845.
Rosenthal, R. (1991). Meta-analytic procedures for social
research (revised). Newbury Park, CA: Sage.
Ruff, H. & Rothbart, M. (1996). Attention in early development: Themes and variations. New York: Oxford
University Press.
Schweizer, K., Moosbrugger, H. & Goldhammer, F.
(2005). The structure of the relationship
between attention and intelligence. Intelligence,
33, 589–611.
Semple, R.J., Lee, J., Rosa, D. & Miller, L.F. (2010).
A randomised trial of mindfulness-based cognitive therapy for children: Promoting mindful
attention to enhance social-emotional resiliency
in children. Journal of Child and Family Studies, 19,
218–229.
Steele, A., Karmiloff-Smith, A., Cornish, K. & Scerif,
G. (2012). The multiple subfunctions of attention: differential developmental gateways to
literacy and numeracy. Child Development, 83(6),
2028–2041.
Strauss, M.E., Thompson, P., Adams, N.L., Redline, S.
& Burant, C. (2000). Evaluation of a model of
attention with confirmatory factor analysis.
Neuropsychology, 14(2), 201–208.
Stuss, D.T., Shallice, T., Alexander, M.P. & Picton,
T.W. (1995). A multidisciplinary approach to
anterior attentional functions. Annals of the
New York Academy of Science, 769, 191–211.
Tang, Y.Y. & Posner, M.I. (2009). Attention training
and attention state training. Trends in Cognitive
Sciences, 13(5), 222–227.
Thomas, G.L. (2013). Assignment 1: Special Educational
Needs Co-ordinators’ views regarding the attentional
components most commonly used by mainstream
primary school aged children and the attentional
components that the primary school curriculum makes
most demands on. University of Manchester:
D.Ed.Ch.Psychol. Research Paper 1.
Ullmann, R.K., Sleator, E.K. & Sprague, R.L. (1997).
ADD-H Comprehensive Teacher Rating Scale
(ACTeRS). Champaign, IL: MetriTech, Inc.
Educational & Child Psychology Vol. 33 No. 1 63
Measuring the effectiveness of a mindfulness-based intervention for children’s attentional functioning
van der Oord, S., Bögels, S. M. & Peijnenburg, D.
(2012). The effectiveness of mindfulness training
for children with ADHD and mindful parenting
for their parents. Journal of Child and Family
Studies, 21, 139–147.
Ward, J. (2006). The student’s guide to cognitive neuroscience. Hove: Psychology Press.
Wilcoxon, F. (1945). Individual comparisons by
ranking methods. Biometrics, 1, 80–83.
Williams, M. & Penman, D. (2011). Mindfulness:
A practical guide to finding peace in a frantic world.
London: Piatkus.
64 Educational & Child Psychology Vol. 33 No. 1
George Thomas & Cathy Atkinson
Copyright of Educational & Child Psychology is the property of British Psychological Society
and its content may not be copied or emailed to multiple sites or posted to a listserv without
the copyright holder's express written permission. However, users may print, download, or
email articles for individual use.
Pearls, Pith, and Provocation
Qualitative Health Research
19(10) 1495–1503
© The Author(s) 2009
Reprints and permission: http://www.
sagepub.com/journalsPermissions.nav
DOI: 10.1177/1049732309348500
http://qhr.sagepub.com
Understanding the Ecological Validity
of Neuropsychological Testing Using
an Ethnographic Approach
Deborah Gioia1
Abstract
Neurocognitive impairment is a defining and disabling feature of schizophrenia and other physical disorders. Most of
our understanding about neurocognitive deficits comes from laboratory-based testing in research protocols. There
has been little research using direct behavioral community observation over a prolonged period to understand the
association of daily functioning with cognitive performance. The purpose of this study was to develop an observational
method that could be replicated by researchers interested in viewing cognitive deficits in vivo, and then comparing
this data to laboratory measures to affirm the ecological validity of those measures. The eight-step method explained
here was developed from the targeted ethnographic study of 10 persons with schizophrenia. Obtaining real world
context with this method will help to increase the generalizability of effective cognitive treatments, create improved
interventions for this population, and bring into greater relief the coping and compensatory strategies already used by
individuals to complete daily tasks.
Keywords
ethnography; mental health and illness; neuropsychology
Ecological validity has become an increasingly important
notion in science, particularly when we want to understand how individuals who have one outcome from
neuropsychological (NP) laboratory testing (e.g., low
functioning) seem to have more varied outcomes (e.g.,
high functioning in some tasks, low in others) when they
are in their everyday settings. Ecological validity has been
defined as “the functional and predictive relationship
between the patient’s performance on a set of neuropsychological tests and the patient’s behavior in a variety of
real-world settings” (Lezak, Howieson, & Loring, 2004;
Sbordone, 1996, p. 16). There have been difficulties in
relating NP findings and behavioral laboratory measures
to real-world outcomes. An ethnographic approach to
ecological validity will provide new information about
how cognitive impairments impact the lives of persons
affected with NP-involved disorders (e.g., schizophrenia,
multiple sclerosis) in a more direct way than self-report,
clinician ratings, or simulation of a real-world event like
planning to shop for items to make a cake, rather than
actually making the cake. An eight-step ethnographic
method for assessing ecological validity is the focus of
this article.
There are legitimate and important reasons for pursuing a better understanding of ecological validity of
neuropsychological tests. Previous literature has argued
that what is learned in the laboratory about cognitive
functioning might be too artificial or narrow to make
meaningful predictions about how an individual will
respond in a given situation when they have cognitive
decline or deficit (Groth-Marnat, 2000; Koren, Seidman,
Goldsmith, & Harvey, 2006; Sbordone, 1996). Despite
this, NP tests originally developed to provide information
about diagnosis are being used to answer questions about
everyday functioning, with very little empirical evidence
to support this practice (Chaytor & Schmitter-Edgecombe,
2003). When a lab test is administered, all real-world
context is stripped from the task so as to complete the test
with as few distractions as possible. In daily life, distractions and complications abound with even the simplest
task, so this artificial performance does not tell us much.
We know that individuals with NP deficits demonstrate
much creativity and resiliency in the face of these realworld distractions, but we have not had a good way to
1
University of Maryland, Baltimore, Maryland, USA
Corresponding Author:
Deborah Gioia, University of Maryland, School of Social Work, 525 W.
Redwood St., Baltimore, MD 21201, USA
E-mail: dgioia@ssw.umaryland.edu
1496 Qualitative Health Research 19(10)
capture these data and compare them to NP test results in
a distraction-free environment.
Groth-Marnat (2000) reviewed the qualitative/quantitative continuum of assessment as it relates to the
understanding of the individual client. The author noted
the divide between these assessment approaches with
regard to the external validity of NP tests. He stated,
“Test validity traditionally has not focused on a person’s
everyday abilities/characteristics but rather on such areas
as correlation with previously developed measures, factor
analyses, and level of diagnostic agreement” (GrothMarnat, 2000, p. 357). One hopes that a fuller integration
of testing and behavioral observation, coupled with the
incorporation of emerging technology and theory, and
research discoveries would lead us to a richer understanding of the functioning of the individual in context, and
provide a means to get needed interventions and services
to individuals in the community in an expedited manner
(Bellack et al, 2007; Brekke, Ell, & Palinkas, 2007;
National Institute of Mental Health [NIMH], 2008).
Rather than attribute real-world performance or prediction of future performance to any single neurocognitive
test or proxy measure, which is frequently implied but is
a weak measure of prediction (Sbordone, 1996), this proposed method captures a dimension that has not been
integrated in the ecological validity or prediction literature.
Prior to articulating the steps of the method, a brief
review of ecological validity studies and proxy measures
of NP functioning are described.
Neuropsychological Assessment
and Everyday Life
Chaytor and Schmitter-Edgecombe (2003) reviewed the
empirical studies in which ecological validity of neuropsychological (NP) tests was assessed against individuals’
use of everyday skills and found that the magnitude of
relationships between individual tests and measures of
outcome was weak. The studies reviewed mainly had
populations diagnosed with closed head injuries and
chronic medical conditions (e.g., multiple sclerosis), and
only two studies looked at individuals with schizophrenia.
However, none of the outcome measures used in any of
the studies was a direct behavioral observation of the
person in the community; rather, outcomes relied mainly
on clinician ratings, self- and informant reports, and
questionnaires. This review highlighted that our understanding to date of how persons with cognitive deficits
make decisions and function in their homes, social environments, and service entities is largely unknown.
Illness- and injury-specific studies abound in the NP
assessment literature. In Alzheimer’s research, the ability
to predict self-neglect in community functioning from NP
tests has emerged as promising, but only end-state negative outcomes (e.g., hospitalization or report to adult
protective services) were correlated with tests (Tierney,
Snow, Charles, Moineddin, & Kiss, 2007). A study comparing traumatic brain-injured (TBI) and normal adult
subjects in their self-reported slips of attention (e.g.,
burning the toast, throwing out the vegetables with the
peels) demonstrated in part that there was not a lot of difference between the two groups on a particular neurocognitive
measure of attention and in their self-reported slips of attention, as was hypothesized (Robertson, Manly, Andrade,
Baddeley, & Yiend, 1997). These studies and others illustrate the elusiveness of reliable or standardized predictions
in heterogeneous populations. Without a working knowledge of the actual demands that the environment places on
individuals with or without cognitive impairment, without
an emphasis on and understanding of context, predictions
about community functioning are tenuous, at best. This
overall poor prediction of community abilities based on
any specific NP tests has led researchers to consider other
assessment methodologies, namely proxy measures.
Proxy Approaches to
Neuropsychological Assessment
Thus far, ecological validity in the community has been
assessed with tasks that are completed and rated in office
settings but are crafted as simulations of real-world activities and decisions. However, many of these tasks or
activities assessing an individual’s functioning are limited approximations of real-world tasks. Three examples
of proxy measurement from schizophrenia research are: (a)
the University of California, San Diego Performance-based
Skills Assessment (UPSA; Patterson, Goldman, McKibbin,
Hughs, & Jeste, 2001). This assessment directly examines
participants’ abilities to perform “instrumental” activities of
daily living (e.g., preparing a shopping list, planning a bus
trip), but does so in an office environment and yields a summary score of 0 to 100 to assess overall NP performance.
A second NP proxy assessment outside the clinic setting involves having an individual perform tasks in a
natural setting (e.g., supermarket) using a standardized
rating scale, such as the Test of Grocery Shopping Skills
(TOGSS; Brown, Rempfer, Hamera, & Bothwell, 2006;
Greenwood, Landau, & Wykes, 2005; Rempfer, Hamera,
Brown, & Cromwell, 2003). The grocery trip is organized by the researcher and the actions of the individual
in response to the test directions are rated and quantitatively scored. Following are three examples of proxy
measurement from schizophrenia research. The first is
the University of California, San Diego Performancebased Skills Assessment (UPSA; Patterson, Goldman,
McKibbin, Hughs, & Jeste, 2001).
Gioia 1497
A third approach to ecological validity uses an experience sampling method (ESM) to get closer to cognitive
impairments at the time they occur (Myin-Germeys,
Delespaul, & Van Os, 2005). Participants were given a
programmed digital wristwatch and self-assessment forms
to complete on their thoughts, current context, and mood
when the watch alarm activated. Current iterations of this
method involve using personal digital assistants (PDAs)
to prompt participants to answer particular questions
while going about their activities in the community
(Granholm, Loh, & Swendson, 2008). Although this
method captures a wider breadth of activities than supermarket shopping, and is linked to ongoing community
behavior, it does not provide direct observations of
behavior in natural contexts and precludes direct questioning by the researcher about activities as they occur.
Although proxy measures represent advances in shifting assessment and skill measurement to community
settings, they fall short in the application of direct observation of behavior in the places where individuals live
and work. The eight-step model to capture real-world
functioning presented here is not hypothetical, but one
that has been used in funded National Institute of Mental
Health (NIMH) research to explore ecological validity
(Gioia, 2006; Gioia & Brekke, 2009). The basic features
and steps of the model are presented as explicitly as possible so as to highlight key contributions to make it
accessible and easily replicated in future studies.
Study Information
Summarized information about the study sample and
design from which this ecological validity method was
developed is provided here, and is more fully reported
elsewhere (Gioia, 2006; Gioia & Brekke, 2009). The
present study was approved by the Institutional Review
Board for the University of Southern California and
funded by NIMH. The participants (n = 10) were all
adults, including 5 men and 5 women with schizophrenia
spectrum disorders, and were originally enrolled in a longitudinal study, “Predicting Psychosocial Rehabilitation
Service Outcomes” (n = 140; Brekke, Raine, Ansel,
Lencz, & Bird, 1997), which was the parent study for this
sample. Participants had completed all data collection
with the parent-study protocol prior to being approached
for this study. They all signed new consent forms that
indicated their voluntary participation and the procedures
for protecting their anonymity. Consent procedures for
vulnerable populations used in the parent-study protocol
were followed to ensure comprehension of the form,
including reading the form to the participant when necessary because of limited literacy or lack of reading glasses.
Participants understood that they could withdraw from
the study at any time with no impact to or loss of their
current mental health services. Individuals were paid for
their participation ($20/day), and their decisions about
how to use their study payment became part of the study
data.
The parent-study participants were divided into three
groups based on their NP composite scores (low, midrange, and high functioning). The completed NP tasks
that led to distinguishing these groups included measures
of attention, memory, problem solving, processing speed,
language, and recall, which are typically used in NP batteries for persons with schizophrenia. The mid-range
group (30%) was omitted from the sampling frame so as
to maximize the two extremes of functioning (high vs.
low) for the observation. I was blind to the neurocognitive status of the 10 study participants during the
observations. Participant housing ranged from fully independent apartments, to living with family, to congregate
living, and the length of time in these settings varied.
Because the parent study measured service characteristics, all 10 participants had some current linkage to
community mental health programs, all were taking medication—though some were not fully managing their own
medication schedule, and none were on long-lasting
injectable medication. Complete sampling procedures,
study measures, and sample characteristics from the
parent study are reported elsewhere (Gioia & Brekke,
2009). Although all participants had the option of discontinuing the study at any time, none chose to do so.
The following general points about the 10 individuals
in the study provide some demographic context: (a) All
individuals were on oral antipsychotics and often other
medication, managed by themselves or family members;
(b) none reported abusing substances at the time of the
study, and no toxicology screens were used to corroborate the self-report; (c) all were connected to some level
of community mental health services; (d) all were housed
from independent living to group facilities; (e) all had some
type of regular income (including disability payments),
although all were living in tight financial situations and
had little discretionary income; (f) all had some supportive relationships in their lives—but not necessarily family;
and (g) none were currently in a cognitive remediation
program to remedy their deficits.
The study design was a mixed-method triangulation
convergence design as articulated by Creswell and Plano
Clark (2007). Qualitative and quantitative data were
viewed as having equal importance in the model. The
analysis was an integration of individual participant NP
scores, standardized measures, and ethnographic data;
individual data were analyzed, compared, and contrasted.
Researchers typically use this design when they want to
compare results or to validate, confirm, or corroborate
1498 Qualitative Health Research 19(10)
quantitative results with qualitative findings (Creswell &
Plano Clark, 2007). Although not typically a qualitative
procedure, the decision for this study was to blind the
field interviewer to the high/low NP status of the individuals, thus removing any possible source of researcher
bias that might occur if the cognitive status was known.
Use of the blind was a novel design feature that should be
considered for fit in relation to one’s study. It is not
included in the model steps presented here, but use of the
blind was felt to enhance the rigor of the findings.
The main question for this study was whether there
was a difference in those individuals who had high
and low scores on their laboratory tests and their functioning in the community. Daily tasks and weekly or
monthly tasks were observed within a 3-day period. These
included tasks and activities such as grocery shopping,
eating out (as in a restaurant), doing laundry, scheduling
appointments, and using public transportation. Interest
was not only in observing the tasks as they unfolded
during the follow-along observation in the community;
there was also interest in individuals’ descriptions (upon
being asked) of what they felt about their capabilities/
strategies in completing daily tasks, and their personal
recognition of deficits. These conversations usually took
place at the end of the task (e.g., arriving home from the
market), and were considered part of the individual data.
Steps in Ecological Validity Assessment
Neuropsychological testing can either be completed as a
full battery prior to enrolling in an observation study, or
one can use a brief community-based neuropsychological
assessment such as The Repeatable Battery of the Assessment of Neuropsychological Status (RBANS; Randolph,
1998), which can be administered at the start of the study
in a community setting, and which takes approximately
30 minutes to complete. Test results from either type of
battery can be calculated as a single summary score to
predetermine cutoff ranges for high and low NP functioning levels in the analysis of the qualitative data.
The following eight steps were found to be useful in
the present study (Gioia, 2006; Gioia & Brekke, 2009),
which was designed for this small sample but has applicability for subsamples of much larger studies.
Many of the steps of the model were identified in the
design phase, whereas others were adopted during data
collection. Although the method is similar in many ways
to participant observation, the difference is that the intent
of the observations is always to follow the cognitive strategies the individual employs in relation to how they
perceive their illness deficits. This requires that the field
researcher has background and training in recognizing
basic cognitive deficits and inquiring when they are displayed. A descriptive guide to the steps in the model is
presented.
Step 1: Getting to Know the Participant
This critical step makes use of the initial time spent getting to know the individual at their service site, by getting
to know the individual’s routines through the gathering of
basic demographic, symptom, and psychosocial data (the
Table 1. Eight Steps in the Ecological Validity Process
Step
1. Getting to know the participant
2. Utilizing prolonged engagement with participants
3. Encouraging participant-led activities with opportunity
for reflection
4. Taking field notes
5. Having contact with significant others and community
members
6. Coding and categorizing observed behaviors
and strategies
7. Employing intercoder agreement as a means to
strengthen validity of the data
8. Using the cocreated results
Researcher Task
Develop rapport, collect quantitative data, schedule community
observation
Spend 3 consecutive days viewing typical and atypical tasks
Build in time for participant reflection on use of metacognitive
processes
Write notes in a timely manner at the end of each field day
Check for gaps in understanding with members of team and return
to field to check understanding with participants
Determine depth of interaction with other individuals, as they can
provide collateral information on functioning
Use IADL language to create categories that can be used as a
template for additional coders to rate
Use multiple coders for field notes and have them code notes
according to types of tasks
Generate intercoder reliability scores
Results should be checked with the participants for accuracy
Results should be shared with scientists working in neuropsychology to increase understanding of ecological validity
Gioia 1499
quantitative data; see Gioia & Brekke, 2009). This step
occurs after consent to participate in the study has been
obtained. During this rapport-building phase, a plan for
the researcher to join the individual in their community is
cocreated; this is not a simple process and might be taxing
for the participant. Carlsson, Paterson, Scott-Findlay,
Ehnfors, and Ehrenberg (2007) discuss multiple strategies
for interviewing participants who have communication
and cognitive difficulties (e.g., recognizing fatigue). The
researcher needs to take ample notes as the potential participant describes what they will be doing over the next
few days. If there is a generic activity that the researcher
wants to observe with each person (e.g., money management), and the person has not mentioned this activity,
these areas will be raised to see when this activity might
next come up for them (e.g., when their monthly check
arrives). Observations should be scheduled in the least
obtrusive way possible. Evidence for rapport-building
would be demonstrated by the participant’s trust in the
initial process and by their adequate communication
skills. There is no formal measure for this kind of engagement. Exclusion would be based on high levels of
psychiatric symptoms that interfere with rapport (e.g.,
manic or highly anxious behavior, extreme paranoia). In
this initial visit it is also critical to have the researcher ask
the individual how he or she wants to introduce the
researcher to others as he or she accompanies them in the
community. The introduction might take one form for
close personal friends and family, and another for neighbors and mental health professionals. Rehearsing these
introductions might be helpful.
Step 2: Utilizing Prolonged Engagement
With Participants
To create credible findings for ecological validity of NP
assessments, one needs to spend ample time with each
individual to allow for opportunities to view a range of
community activities, some of them repeatedly. The
method employed here is somewhere between rapid ethnographic assessment and a full ethnography. The rapid
ethnographic assessment is a technique first developed
by Scrimshaw, Carballo, Ramos, and Blair (1991) in HIV
studies to yield contextual data in a short period of time
to obtain information from participants that will likely be
explored in a full qualitative study. A full ethnography
involves a thorough cultural immersion that involves
staying in people’s lives for an extended period of time,
usually months to years (Hammersley & Atkinson, 1995).
The method presented here calls for a period of initial
immersion (9 to 15 hours over 3 consecutive days,
approximately 3 to 5 hours per day; weekends and evenings are possible) viewing participant-led activities, and
most importantly, honing in on specific tasks that are part
of instrumental activities of daily living (IADLs; Lawton
& Brody, 1969). IADLs are activities such as using the
telephone, getting to places beyond walking distance (use
of transportation), grocery shopping, meal preparation,
household chores, laundry, taking medications, and managing money. The IADLs form the foundation of observation
for all populations with cognitive decline, and thus serve
as a baseline of common tasks for analysis. In addition,
each individual has her or his own interests (e.g., using
computers, crafts and model making, attending church),
which provide additional opportunities to view a wide
range of complex tasks beyond IADLs. Every opportunity should be made to attend these additional activities,
even though some might be out of the immediate community in which the participant lives. Activities that
cannot be viewed in the initial 3 consecutive days, or
those needing additional observation, will be scheduled
with the participant when they naturally occur.
Step 3: Encouraging Participant-Led Activities
With Opportunity for Reflection
Once the observation of the participant begins there are
many opportunities to use questions during the task or
after the completion of a complex task—such as ordering
a meal in a restaurant—to probe some of the specific task
components. For example, one individual in the study
ordered a meal in a fast food restaurant and chose a milkshake that he did not like, which created a dilemma for
him. Questions posed by the researcher at the moment of
the event offered the individual a chance to think aloud
with the researcher listening, and decide what to do next
(e.g., keep the shake or request a new one). This example
of an event offered a distinct opportunity to view problem
solving as it unfolded. The technique is similar to that
used in cognitive interviewing when individuals are commenting on the development of a measure (Willis, 2005).
By observing events in real time, there is an increased
opportunity to discover how the person creates a plan of
action around the various tasks, to witness their emotional
state while doing the task, as well as to examine their
decision making about the particular action. Metacognitive processes are often called “thoughts about thinking,”
and can enhance the observed experience in ways not
tapped by observation or tests alone (Lysaker et al,
2005). Obtaining reflection from the participant serves
as a clarification about what the interviewer observed
and also serves as a validity check of the method.
Questions about awareness of deficits in thought or
action can be asked. Enough information needs to be
solicited to determine whether any struggles observed are
because of a social emotional context (anxiety, fear) or
1500 Qualitative Health Research 19(10)
whether it was difficult for the individual to conceptualize the skills necessary to negotiate the task (e.g., talking
with a waiter about returning an incorrect order). The
model presented here captures both the strengths and
difficulties of the individual with neurocognitive deficits
in the community context.
Step 4: Taking Field Notes
The least obtrusive way to collect data about everyday
activities is through the use of field observation and
notes. Wolcott (1995), in The Art of Fieldwork, provided
suggestions to help the researcher “be there” in the field.
He does not recommend that the observer be aloof, but
rather that the observer constantly challenge him- or herself about the observed behavior, which is a great fit for
observing and understanding cognitive deficits. The strategies of field work are mainly nondirective, but there is a
flow between asking direct questions and asking no questions at all, which feeds into the art of this procedure.
There is no absolute way to conduct a field study (van
Manen, 1988).
As soon as possible after the observation is completed,
the observer should write up notes that describe the
events in abundant detail, which will refresh your
memory when you return to the notes. The write-up
should occur prior to the next scheduled observation,
otherwise events will blur and crucial details might be
lost. Observed events directly related to NP should be
listed and taken with the researcher back to the field, to
assist in understanding whether the observation was isolated or more frequently occurring. Consultation with a
neuropsychologist is useful in this stage, as well as in the
ongoing data analysis. Digital audio recorders used postinterview can help to get main thoughts down quickly,
and details can be added when doing the full write-up of
the observation.
Step 5: Having Contact With Significant Others
and Community Members
During the observation it is likely that the individual will
have social contact with others in her or his community,
which is an important area of functioning to observe.
Mental health professionals might be included in these
community contacts. The additional information from
these significant others could prove useful as collaborative information, especially in understanding their role in
helping the individual with daily tasks. These persons can
serve as primary resources for the individual completing
their daily routine tasks. They can provide an additional
source of data to triangulate; however, this would require
a separate set of permission/consent forms. It is important
to negotiate with the individual in the study how these
transactions should be made.
Step 6: Coding and Categorizing Observed
Behaviors and Strategies
One key use of ecological validity is as an assessment
tool, because the integrated data collected can be used to
make determinations or predictions about functional outcomes. One rubric for analysis is to use predominant
behavioral strategies as used in IADLs for describing
observed tasks. In this study, in developing the ecological validity method, four types of strategies were used:
(a) instrumental, (b) rote, (c) social facilitative, and
(d) social independent. Although there is overlap in these
strategies, the analysis called for making an overall judgment about the predominating behavioral style.
Rote activities enable basic community living (such as
doing chores), but are largely characterized by repetitious
or scripted behavior that is often, but not always, demonstrated and guided by others. Instrumental tasks are most
aligned with multiple activities of daily living—self-care,
shopping and other activities related to food, making and
keeping appointments, keeping clothes clean—and are
generally more complex, goal-oriented, and a bit more
self-directed. Facilitative social activities are social connections formed with people to facilitate daily functioning,
usually of the more complex instrumental tasks (e.g.,
asking someone to help with budgeting, with transportation, or advice about decisions). Independent social
activities are social connections and interactions that are
actively cultivated by the individual and that enrich work,
school, living, community, and family tasks, but the individual is not largely dependent on others for completion
of tasks or for direction (Gioia & Brekke, 2009). Other
rubrics or templates for analysis are likely to be developed with the addition of more individual observations.
Step 7: Employing Intercoder Agreement as a
Means to Strengthen Validity of the Data
Participant profiles are created from the descriptions of
tasks in the field notes as a means to reduce the data into
a document that can be read, coded, and rated by multiple
coders. These are basically summary narratives of daily
activities created by the researcher for rating purposes;
summary narratives were created in the current study of
10 individuals (Gioia & Brekke, 2009). After the initial
coding was completed by the author, instructions were
given to a second coder to make judgments about the fit
or lack of fit with the behavioral styles mentioned above.
This analytic task effectively quantitized the qualitative
data into categories that could be compared across coders.
Gioia 1501
Using this approach, Gioia and Brekke (2009) found that
having two coders independently make inferences from
the text was a useful approach for achieving credibility of
the data. Cohen’s (1960) Kappa is a statistical formula
that takes into account the relationship between the proportion of times when the raters agree on the data and the
proportion of times when agreement might have been
expected by chance. The level of agreement in this study
was .77, which meets the benchmark for achieving a substantial level of agreement between coders. Future
iterations of this model should be able to incorporate
multiple coders making independent ratings of the field
notes from a greater number of study participants on
these and other dimensions of interest to the study. It
might be possible to include the mental health clinician
who works with the individual to verify the predominant
behavioral style. This was not included in this study but
would provide an excellent additional means of study
rigor in future research. This level of analysis is complex,
and the relationship between neurocognition and behavior in the open environment can be moderated by many
contextual variables, including other physical or mental
illnesses, which will need to be considered from the qualitative findings in the definition of cognitive impairment
(Franzen & Wilhelm, 1996).
Step 8: Using the Cocreated Results
Results should optimally be cocreated and checked with
the participant living in the community. In other words,
validating that the strategy used to offset the neurocognitive
deficit was one that was agreed on by both the researcher
and the participant. Fully employing a participatory
action research (PAR) model into the eight steps would
render the findings clinically useful as a means to educate,
empower, and engage the individual, their support
network, the researcher, and the research audience about
the challenges to perform daily tasks in the community
when one has a disorder that affects cognition (Kidd &
Kral, 2005). Persons with neurocognitive impairment are
typically cut off from PAR engagement because of stigma
and other beliefs about their ability to participate.
However, this research method of ecological validity
could be ideal for consciousness raising for stakeholders,
and contributes an essential understanding of the notions
of ecological validity in these NP disorders.
Discussion
Although there has been a tendency to focus on negative
aspects of cognitive deficits that result from injuries and
illnesses, we know that some individuals with NP deficits
do not necessarily remain at an impaired level when they
are in community settings (Gioia, 2006; Koren et al,
2006). Why this happens for some and not for others is at
the heart of an ecological validity method. A step-bystep process was outlined to operationalize a method for
enhancing ecological validity by utilizing communitybased observation methods to assess whether there was
any correlation between NP functioning in the laboratory
and real-world functioning. Although eight steps have
been described, there is likely a way to tailor this approach
for use in a variety of projects. Step 8 is recommended but
could be omitted. The other steps are optimal for producing data on ecological validity. The proposed model of
behavioral community observation of cognition and tasks
has been suggested as a next step by many researchers.
The eight-step method appears to be relatively
straightforward, with the exception of the prolonged time
commitment, but it is deceptively complex, and presents
difficulties that are inherent to the fieldwork process and
the population being studied. It also might be difficult
because of the culture, gender, and ages of the participant
and the field researcher. Some of the eight steps might be
problematic for individuals with schizophrenia, and in
fact, in this sample, 10 individuals who had paucity of
speech or ongoing psychotic symptoms required accommodations to the schedule. One individual who became
paranoid during our research time in the community
required shortening the session by one day, but we were
able to resume at a later time. Research decisions might
have to be made to maximize the ability of the participant
to comfortably engage in the process, and yet the reality
of a prolonged engagement is likely to tire individuals
and risk the possibility of prematurely ending the participant observation. It might also be embarrassing for the
individual to have someone accompany her or him everywhere for a few days, or there might be times when safety
in the community is a concern for the researcher and the
participant. These potential community interactions can
be viewed as data about the process of obtaining ecological validity, and will certainly assist in refining the steps
of the proposed method.
Cognitive difficulties are a significant factor in many
chronic mental and physical conditions, such as Alzheimer’s disease, multiple sclerosis, schizophrenia, Parkinson’s
disease, cardiac conditions, traumatic brain injury, substance use, and normal aging. A limitation of this and
other NP studies is that there are often multiple contributions to cognitive decline which cannot really be separated
at the time of the data collection (e.g., silent heart disease). Most neuropsychologists continue with their studies
despite this real-world messiness that creates interpretation difficulties. It is hoped that this proposed model of
measuring ecological validity might be used across populations, and might provide a more attenuated description
1502 Qualitative Health Research 19(10)
of disparate capacities within and between individuals
with the same disorders. It is important that future
researchers exploring NP consider not only laboratory or
proxy findings, but provide a view of the individual as he
or she functions in his or her own environment. Even
though this initial study was mainly concerned with the
functional aspect of ecological validity—and not the predictive capacity—refinement of this method with more
participants might yield a greater ability to predict how
well individuals will be able to attend to routine and nonroutine tasks with and without social assistance.
The triangulation of observational and NP test data
should shed light on the differences in and types of cognitive decline, as well as differences in and types of
strategies across populations and illnesses. It might tease
out the question of whether the same neuropsychological
test is as relevant for someone with schizophrenia as it is
for someone with traumatic brain injury. In other words,
are designations of high and low functioning from the lab
results telling us the same thing, or do specific illness
characteristics play a larger part in the outcome? Individuals collecting the observational data would need
specific procedural training in field methods to ensure
that probes concerning the tasks are similar across field
researchers.
One possibility for a future study would be to have
four types of data collected for each individual within a
single study: (a) scores from NP laboratory measures; (b) a
proxy measure such as self-report, use of PDAs, or performance in one of the coprimary measures of NP (Green
et al., 2008); (c) significant other/mental health professional reports; and (d) the prolonged participant field
observation described here. Although this level of analysis will have complex challenges, it is promising to be
able to utilize all known methods for understanding NP
capacity and thus increase the potential for predictions of
community functioning and subsequent matching with
treatment interventions. It is hoped that this information
will contribute to efforts aimed at restoring or preserving
individual cognitive functioning. Cognitive remediation
is a current area of research that uses an array of behavioral interventions, mainly computer programs and other
targeted learning tools, to enable individuals to retrieve
some of their lost cognitive skills by teaching a variety of
compensatory strategies (Medalia & Richardson, 2005).
Four additional considerations need to become part of
the evaluation of an ecological validity model in the
future: (a) Can the steps of the model be accomplished in
a timely manner and still produce the complexity of data
necessary for a rigorous analysis?; (b) Can this ecological
validity approach be used to map the actual daily performance of slightly larger samples (n = 25), or subsamples
of much larger studies?; (c) What is the cost/benefit of
this ecological validity approach as compared to other
means of collecting information on NP functioning, and
what does it add?; and (d) Will this data be used to
develop interventions that benefit the recovery of the
individual in the community?
Author’s Note
I extend special thanks to the study participants for allowing me
to be part of their daily lives in the community, and to Dr. John
Brekke for his research guidance and mentorship.
Declaration of Conflicting Interests
The author declared no conflicts of interest with respect to the
authorship and/or publication of this article.
Funding
The author disclosed receipt of the following financial support
for the research and/or authorship of this article: This research
was supported by the National Institute of Mental Health (R03
MH64686-01A1).
References
Bellack, A. S., Green, M. F., Cook, J. A., Fenton, W., Harvey,
P. D., Heaton, R. K., et al. (2007). Assessment of community functioning in people with schizophrenia and
other severe mental illnesses: A white paper based on a
National Institute of Mental Health sponsored workshop.
Schizophrenia Bulletin, 33(3), 805-822.
Brekke, J. S., Ell, K., Palinkas, L. A. (2007). Translational science at the National Institute of Mental Health: Can social
work take its rightful place? Research on Social Work Practice,
17(1), 123-133.
Brekke, J. S., Raine, A., Ansel, M., Lencz, T., & Bird, L. (1997).
Neuropsychological and psychophysiological correlates of
psychosocial functioning in schizophrenia. Schizophrenia
Bulletin, 23(1), 19-28.
Brown, C. E., Rempfer, M. V., Hamera, E., & Bothwell, R.
(2006). Knowledge of grocery shopping skills as a mediator
of cognition and performance. Psychiatric Services, 57(4),
573-575.
Carlsson, E., Paterson, B. L., Scott-Findlay, S., Ehnfors, M. &
Ehrenberg, A. (2007). Methodological issues involving people with communication impairments after acquired brain
damage. Qualitative Health Research, 17, 1361-1371.
Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecological validity of neuropsychological tests: A review of the
literature on everyday cognitive skills. Neuropsychology
Review, 13(4), 181-197.
Cohen, J. (1960). A coefficient of agreement for nominal scales.
Educational and Psychological Measurement, 20(1), 37-46.
Creswell, J. W., & Plano Clark, V. L. (2007). Designing and
conducting mixed methods research. Thousand Oaks, CA:
Sage.
Gioia 1503
Franzen, M. D., & Wilhelm, K. L. (1996). Conceptual foundations of ecological validity in neuropsychological assessment. In R. J. Sbordone & C. J. Long (Eds.), Ecological
validity of neuropsychological testing. Delray Beach, FL:
GR Press/St Lucie Press.
Gioia, D. (2006). A contextual study of daily living strategies
in neurocognitively-impaired adults with schizophrenia.
Qualitative Health Research, 16, 1217-1235.
Gioia, D., & Brekke, J. S. (2009). Neurocognition, ecological
validity, and daily living in the community for individuals
with schizophrenia: A mixed methods study. Psychiatry:
Interpersonal and Biological Processes, 72(1), 93-106.
Granholm, E., Loh, C., & Swendson, J. (2008). Feasibility
and validity of computerized ecological momentary assessment in schizophrenia. Schizophrenia Bulletin, 34(3), 507-514.
Green, M. F., Nuechterlein, K. H., Kern, R. S., Baade, L. E., Fenton, W. S., Gold, J. M., et al. (2008). Functional co-primary
measures for clinical trials in schizophrenia: Results from the
Measurement and Treatment Research to Improve Cognition
in Schizophrenia (MATRICS) psychometric and standardization study. American Journal of Psychiatry, 165, 221-228.
Greenwood, K. F., Landau, S., & Wykes, T. (2005). Negative
symptoms and specific cognitive impairments as combined
targets for improved functional outcome with cognitive
remediation therapy. Schizophrenia Bulletin, 31(4), 910-921.
Groth-Marnat, G. (2000). Visions of clinical assessment: Then,
now, and a brief history of the future. Journal of Clinical
Psychology, 56(3), 349-365.
Hammersley, M., & Atkinson, P. (1995). Ethnography: Principles in practice. New York: Routledge.
Kidd, S. A., & Kral, M. J. (2005). Practicing participatory action
research. Journal of Counseling Psychology, 52(2), 187-195.
Koren, D., Seidman, L. J., Goldsmith, M., & Harvey,
P. D. (2006). Real-world cognitive- and metacognitivedysfunction: A new approach for measuring and remediating more “right stuff.” Schizophrenia Bulletin, 32(2), 310-326.
Lawton, M. P., & Brody, E. M. (1969). Assessment of older
people: Self-maintaining and instrumental activities of daily
living. Gerontologist, 9, 179-186.
Lezak, M. D., Howieson, D. B., & Loring, D. W. (2004). Neuropsychological assessment. New York: Oxford University Press.
Lysaker, P. H., Carcione, A., Dimaggio, G., Johannesen, J. K.,
Nicolo, G., Procacci, M., et al. (2005). Metacognition amidst
narratives of self and illness in schizophrenia: Associations
with neurocognition, symptoms, insight and quality of life.
Acta Psychiatrica Scandinavica, 112(1), 64-71.
Medalia, A., & Richardson, R. (2005). What predicts a good
response to cognitive remediation interventions? Schizophrenia Bulletin, 31(4), 942-953.
Myin-Germeys, I., Delespaul, P. H., & Van Os, J. (2005).
Behavioral sensitization to daily life stress in psychosis.
Psychological Medicine, 35(5), 733-741.
National Institute of Mental Health. (2008). National Institute of Mental Health strategic plan (NIH Publication No.
08-6368). Washington, DC: Government Printing Office.
Retrieved August 15, 2009, from http://www.nimh.nih.gov/
about/strategic-planning-reports/index.shtml.
Patterson, T. L., Goldman, S., McKibbin, C. L., Hughs, T.,
& Jeste, D. V. (2001). UCSD performance-based skills
assessment: Development of a new measure of everyday
functioning for severely mentally ill adults. Schizophrenia
Bulletin, 27(2), 235-245.
Randolph, E. T. (1998). Social networks and schizophrenia. In
K. T. Mueser & N. Tarrier (Eds.). Handbook of social functioning in schizophrenia (pp. 238-246). Needham Heights,
MA: Allyn & Bacon.
Rempfer, M. V., Hamera, E. K., Brown, C. E., & Cromwell, R. L.
(2003). The relationship between cognition and the independent living skill of shopping in people with schizophrenia.
Psychiatry Research, 117(2), 103-112.
Robertson, I. H., Manly, T., Andrade, J., Baddeley, B. T. &
Yiend, J. (1997). Oops!: Performance correlated of everyday attentional failures in traumatic brain injured and normal subjects. Neuropsychologia, 35(6), 747-758.
Sbordone, R. J (1996). Ecological validity of neuropsychological testing. In R. J. Sbordone & C. J. Long (Eds.), Ecological validity of neuropsychological testing (pp. 15-42).
Delray Beach, FL: GR Press/St Lucie Press.
Scrimshaw, S. C. M., Carballo, M., Ramos, L., & Blair, B. A.
(1991). The AIDS rapid anthropological assessment procedures: A tool for health education and planning. Health Education & Behavior, 18(1), 111-123.
Tierney, M. C., Snow, W. G., Charles, J., Moineddin, R., &
Kiss, A. (2007). Neuropsychological predictors of selfneglect in cognitively-impaired older people who live
alone. American Journal of Geriatric Psychiatry, 15(2),
140-148.
van Manen, J. V. (1988). Tales of the field: On writing ethnography. Chicago: University of Chicago Press.
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.
Wolcott, H. F. (1995). The art of fieldwork. Lanham, MD:
AltaMira Press.
Bio
Deborah Gioia, PhD, LCSW, is an associate professor at the
University of Maryland, Baltimore, School of Social Work,
Baltimore, Maryland, USA.