Funded over three years by the Commonwealth Government’s Stronger Families & Communities Invest to Grow strategy, the Through the Looking Glass (TtLG) Project is a health, education and welfare collaborative early intervention strategy that utilises the existing infrastructure and universality of five childcare settings (sites) across Australia to intervene with families where there is an identified compromised attachment relationship between the parent and child/children.
The project’s goal is to ‘develop and pilot a model of collaborative early intervention and prevention for targeted parents to improve secure attachment outcomes for young children’. The project is designed to achieve specific outcomes for parents (targeting mothers), children and childcare staff and to develop and promote a ‘best practice’ service model to address issues of attachment. The Lady Gowrie Management Team worked collaboratively with the Evaluator to specify project objectives/ desired outcomes. These were ordered hierarchically and an evaluation plan designed.
The TtLG Project provides intensive psychosocial support, therapeutic intervention and childcare as a package for high-risk families in order to develop and support secure attachment relationships between mother and child. The primary target group is mothers of children aged 0–5 years. The participating families come from diverse backgrounds but all exhibit multiple risk factors including anxiety, depression and social isolation and many of the parents have reported early trauma in their own lives. There were six implementations of the project (referred to as Waves) conducted, each lasting around five months. There were up to seven families recruited for each of the five sites per Wave. Each childcare centre site provided a clinician, co-facilitator and primary care giver to work with each recruited family. The project was guided by a Reference Group which provided expertise in psychiatry, psychology, education and childcare and acted as a ‘critical reference group’ for the evaluation (Wadsworth 1998). The project was managed by the Lady Gowrie Child Centre in Thebarton, Adelaide.
The TtLG Project is based on Attachment Theory. The intervention draws from the Circle of Security (COS) Project Model (Marvin et al, 2002) which assists parents and childcare staff to understand and integrate Attachment Theory into practice.
This report presents evaluation findings from all six Waves of the TtLG Project.
Evaluation Methodology and Procedures
The evaluation embraced a mixed methods paradigmatically pragmatic stance (Greene and Caracelli, 1977) selecting appropriate evaluation methods for the collection of data to address the range of identified indicators and meet the needs of stakeholders. The evaluation draws on Patton’s (1997) Utilisation-focused Evaluation using participatory action research procedures (Wadsworth 1998: Sankaran et al, 2001).The evaluation also adheres to the tenets of ‘Realistic Evaluation’ (Pawson & Tilley 1998).
The evaluation, designed and managed by the External Evaluator has utilised an Evaluation Assistant position based at Lady Gowrie, Thebarton to enhance the integration of evaluation procedures and on-going feedback into project practice. Capacity training in evaluation for all project staff was on-going, and formal evaluation feedback was facilitated through the Reference Group which met regularly with the Evaluator throughout the project. Formal feedback was provided approximating to the completion of each Wave to facilitate discussion and planning for project improvement (evaluation was a standing agenda item for each Reference Group meeting). The evaluation also sought to identify areas where the project did not appear to provide benefits for recruited families or problematic issues in its delivery. A formal Interim Evaluation Report including recommendations for project development was provided in July 2007. Evaluation findings were also disseminated across all the project sites throughout the project and presented at a formal gathering of staff in Adelaide on the conclusion of Wave 5.
Evaluation data was obtained through applying a multi-facetted and methodologically triangulated approach. The approach has been flexible in order to adapt to the evolving project and to minimise disruption to the busy workloads of those professionals approached to participate.
In consultation with Lady Gowrie Management, the evaluators established ethically appropriate systems to collect, compile and transfer confidential data from each site to a central point for analysis. This included allocating unique case identifiers to each family, child, site and Wave in order to link the various pre and post data sets. Clinicians were engaged to help collect client data and were provided with client consent forms, evaluation information and summary evaluation sheets for each to complete.
The evaluation has applied a series pre and post-project measurement tools (see: Appendix D), surveys, interviews and observations to collect quantitative and qualitative data from mothers, children and TtLG staff. The Reference Group was requested to identify appropriated standardised instruments to measure a range of psychological and behavioural dimensions related to attachment. No one instrument operationalised the multifaceted nature of attachment and related issues addressed in the project and so a suite of tests was subsequently adopted. Two specific tools addressing ‘wellbeing and involvement’ (child behavioural dimensions) and ‘emotional availability’ (dimensions operationalised through observations of child-parent interactions) necessitated the use of video recording and further staff training to facilitate this. The latter tool required enlisting professional external rating of videos by assessors based in Sydney. For this measure, pre and post project videos of engaged parents’ interactions with their children following each Wave were transported from each site to Sydney and ratings sent back to Adelaide to be included in the analysis. In addressing mothers’ well-being it was hypothesised that the project would improve levels of parental stress, anxiety and depression; specific standardized tools were selected to measure these concepts. Supplemental data from fathers was obtained by self-completion questionnaire.
Sustained impacts for targeted families over the medium term were addressed through follow-up surveys of all mothers from the first three Waves three months after completing the program. Sustained outcomes over a longer time period were addressed through a follow-up survey of Wave 2 and Wave 3 families 16–18 months after completing the project. These Waves were selected as the project had quickly reached a state of maturity following Wave 1, and provided a practical benefit of addressing longer term impact assessment within the three year funding period of the project.
The evaluation also addressed all professional stakeholders who had engaged with the project. Experiences of project management, site managers, project clinicians and project co-facilitators have been addressed throughout the project through representations on the Reference Group and formal email surveys. However, as part of refining the service model, the Evaluator conducted a series of semi-structured interviews with the professional staff across all five project sites. To supplement these two focus groups of site staff who were not directly involved with the project were also conducted to explore the extent to which working practices across the childcare centres had been influenced by the project. Moreover, the need for the evaluation to acquire a broader conceptual understanding of how the project operates ‘in situ’ was identified and an additional ‘quasi ethnographic’ method ‘rapid reconnaissance’ was applied at selected sites.
The Reference Group, Project Management, Childcare Centre CEOs/Directors and TtLG staff worked in partnership to develop and support the TtLG Model. A range of suitable partnering agencies engaged with the project and were committed to it. A number of logistical issues have arisen in the implementation of the project. The mechanisms for identifying and addressing these issues were established through the Reference Group, on-going evaluation feedback, liaison between the sites and the Project Manager and through formal training and information exchange sessions which have been well received.
The TtLG Project has been very active in providing a range of capacity building activities to staff across the five project sites. This has developed skills to adopt and deliver the integrated primary care giving system, which in turn supports the TtLG families and improves attachment outcomes. This has allowed the organisation to deliver better services for
families and their children, (a national Invest to Grow priority).
The project recruited 138 families and 126 of these completed the five month program (a client retention rate of 91.3%). Of the twelve mothers who did not complete the program, four failed to enrol and eight left for reasons which were not related to the project or to any of the demographic variables considered in the evaluation. Additionally, logistical difficulties prevented two sites from fully distributing the post project survey to all participating mothers in Wave 1; survey data was completed by 118 mothers (a response rate of 93.7%).
Formative evaluation has revealed that mothers and fathers have been very positive about their experiences with the project and these feelings continued after completion. Mothers enjoyed the sessions provided and felt comfortable to freely explore their parenting and attachment issues with project staff.
Over nine in 10 mothers felt relaxed and safe at the childcare centre they attended, they felt comfortable with the workers and found it easy to get along with their primary care giver. Mothers clearly enjoyed the weekly sessions and the childcare and primary care giving ethos of the centres were also highly valued. Very few found the sessions difficult to attend. Mothers who attended the project continued to enthuse about it long after finishing.
Given the multifaceted and holistic approach adopted in the Project Model, it is difficult to identify the most important factors which facilitated improved impacts. More than eight in 10 mothers indicated that 80% of the strategies employed had helped them (with six in 10 indicating that 70% of strategies had helped them ‘a lot’) with regard to understanding their child’s attachment needs.
The combinations of group and individual work with clinicians and reflections on the child-parent videotapes guided by insights from Attachment Theory and the Circle of Security have clearly contributed to greater understanding of attachment.
Impact and Outcome Evaluation
91.5% of mothers surveyed indicated that the project had helped them to feel closer to their child, with over half of these feeling this strongly. Over eight in 10 indicated the project had helped them to feel good about themselves as parents. 76% were more confident to look for other services and supports for their family. Nearly all mothers surveyed (97.5%) indicated that they had learnt more about parenting and attachment, with over nine in ten being more confident to respond to their child’s needs, feeling better able to cope as a parent, feeling closer to their child and acquiring more understanding of their child’s attachment and exploration needs.
For many mothers these impacts have been sustained. 88% of mothers noted lasting positive changes in themselves since completing the project. Between eight to nine in ten mothers reported sustained benefits for their parenting practices, being better able to identify and read cues and respond to their child’s needs. 74% reported getting less frustrated with their child three months after completing the project. All of the mothers who were surveyed 16–18 months after completing the project continued to apply learning and skills acquired through it.
Around eight in ten mothers formed supportive friendships during the project with over half of the mothers engaged maintaining friendships three months after project completion. While this reduced over time, 28% of mothers indicated they had retained friendships 16–18 months after completing the project.
While fathers as a group have been engaged in several sites this has tended to be more focused on information giving activities. This has been more advanced in the longer established Thebarton site where group sessions with a number of fathers have included video activities. Many of the mothers who engaged with the project did not have a male partner, and the logistics of assembling working fathers at convenient times have been prohibitive. Those fathers who have engaged in formal group sessions have benefited in terms of raised understanding of attachment and subsequent parental improvements.
A variety of data sources support positive changes in child behaviour attributable to the project. Over eight in ten mothers reported improved positive child behaviour changes. This impact appears to be lasting with 88% of mothers surveyed in the three month follow-up affirming this; it is likely that this has been reinforced by more positive parenting practices. 16–18 months after the project, mothers reported sustained improvement in family functioning; many felt they could do more with their children, enjoyed parenting, were better able to cope and felt they were better parents as a result of the project.
The above findings have been supported through accounts of fathers, and professional stakeholders (both those directly involved with the project and concomitant staff at the childcare centres), and are further supported by the applied pre and post- standardised tools.
Psychological and behavioural improvements were found to be statistically significant in nine of the 11 dimensions measured, with large effect sizes found for reductions in mothers’ depression, anxiety and parental stress levels, and improvements in the child’s wellbeing and involvement observation ratings. Over the duration of the project, the number of mothers experiencing ‘moderate’ to ‘severe’ anxiety and depression (scoring between 11–21 on the Hospital Anxiety & Depression Scale (HADS)), more than halved; from 68 to 33 (anxiety) and from 43 to 13 (depression). Conversely the numbers acquiring a ‘normal’ score (from 0–7) more than doubled for anxiety (from 27 in the pre measure to 53 in the post) and was over 50% higher for depression (from 57 to 87 respectively). Positive impacts in these areas were also found for more ‘marginalised’ mothers with low education levels and receiving state benefits as their household’s main income source. While numbers are small, those from non-Australian backgrounds were over-represented among those experiencing persistent moderate-severe anxiety and depression. The effectiveness of the project for these specific families requires more investigation.
Significant improvements with moderate to large effect sizes were found in the four ‘emotional availability’ (EA) domains hypothesised to improve through the project: Child Responsiveness to Parent, Child Involvement with Parent, Parent Sensitivity and Parent Structuring. The two remaining EA domains Parent Non-Intrusiveness and Parent Non-Hostility, did not demonstrate significant improvement and had an effect size less than ‘small’ using Cohen’s criteria. However, few parents performed below the optimum level in the pre-measure of these two domains and there was general improvement for the small number who did. There was also evidence of a minority of cases performing poorer in the post measure for Parent Non-Hostility. Nonetheless these cases experienced a range of other positive impacts from the project. Conclusions cannot be drawn about the effect of the project in these two EA areas, but it is recommended that further research be conducted in the Non-Hostility domain particularly for University educated mothers and those from culturally and linguistically diverse backgrounds.
Parents have increased their knowledge competence and awareness to overcome barriers to attachment, are less stressed, depressed and anxious and better able to cope as parents. Many report better parenting practices, better engagement with their children and improved child behaviours which they attribute to the project. For many parents these impacts have been sustained over a relatively long period of time since leaving the project. These findings provide clear evidence that the project is addressing the national Invest to Grow priority areas of: Improved family functioning, Improved parent competence and style and Improved child social and emotional development. The few families who did not benefitted in relation to one indicator or aspect, invariably benefited in a range of other areas and measures.
Development of Service Model
A range of issues have been identified with regard to optimising the implementation of the Model. Establishing a ‘primary care giving culture’ and broader understanding of Attachment Theory require on-going training in a field known to have sub-stantial staff turnover. In several sites this task has been embraced as part of the role of the Clinician. Moreover, all the centres and staff engaged with the project have committed to the on-going implementa-tion of primary care giving practice; this is ingrained in policy developments at each centre.
Few culturally and linguistically diverse clients were engaged and this has been largely attributed to the demographics of the site constituencies. Lady Gowrie, Thebarton is currently conducting a separate study investigating the applicability of the Model with Aboriginal communities. This study has utilised funds from the project with the agreement of the funding body and is not part of this evaluation.
Difficulties identified in the Interim Report concerning the adoption of multi-disciplinary team working have been addressed and appear to have largely been resolved within each site. This has occurred as project staff gained mutual appreciation of the expertise each discipline brought to the project. However, a number of areas of potential refinement relating to this were identified; these are detailed in Section 4 of this report. These should generally be viewed as considerations for those seeking to implement the Model rather than stipulations as there will inevitably be contextual and staff differences in different site locations.
Establishing a coherent set of working, reporting and accountability procedures across the five engaged sites proved to be highly problematic particularly as each childcare centre is a sovereign body. This has also been exacerbated by staff turnover and geographical distance, notably with the Perth site which ended its involvement with the project after the fifth Wave. However, all sites (including Perth) have expressed strong wishes to continue with the project in some form.
In the light of the evidence presented through this evaluation, there is an overwhelming case to perpetuate the project in order to build on the investment and continue to provide an intervention which has clear multiple positive impacts and sustainable benefits for Australian families. While there are areas of the service model which may be subject to on-going context specific revision (discussed in section 4 of this report), the project has demonstrated its flexibility to adapt to, and be adopted by different childcare centre practices and contexts and generate a range of successful and profound outcomes for service providers and their clients. However, the need to secure funding for the Clinician and Co-facilitator roles and to support the provision of childcare for project clients is crucially important to the functioning of the project.
Further investigation of the impact of the project for C&LD and Aboriginal families is required before the project can be recommended for these groups. There is also a case for further investigating the relationships of the intervention with the EA domain Parent Non-Hostility particularly for University educated mothers.