Implications of the Evidence for the Ministry of Youth Development

Participation

During the course of the review, it was suggested on more than one occasion that a strengths-based approach meant that all young people should be eligible to participate in all youth development programmes. In respect to MYD youth development programmes, this would mean it was unnecessary to have guidelines that further narrowed the scope of the target group beyond the current Cabinet-mandated 'young people aged between 12 and 24 years'.1 The youth development literature offers an alternative view and suggests our understanding and application of the concept of 'strengths-based' work needs to be more nuanced.

A strengths-based approach certainly does emphasise the importance of seeing the potential in young people and supporting their positive development; without doubt this is of critical value to all young people and not just those deemed 'at risk'. At the same time, there is a substantial body of evidence that tells us that some young people are statistically more likely to be at risk of a negative life trajectory because of a lack of developmental opportunities brought about by a lack of attachment to pro-social settings and people. A 'negative life course trajectory' can represent serious harm and cost not only to the person involved but also to the state and to society generally.

A strengths-based approach does not mean we are blind to this information or this potential risk. We don’t assume that everyone who is statistically at risk is actually at risk, and equally, we don’t assume that everyone who is 'statistically blessed' is firmly set on a positive life trajectory. But the reality is that many young people can access developmental opportunities independent of a formal youth development programme or as part of a low level programme that forms only a small part of their normal daily life. There are programmes that MYD already funds that fall into this category, such as mentoring, Stage Challenge, Scouting NZ, Enterprise NZ and so forth. MYD’s funding pool for structured programmes is relatively small: while this does not dictate who should participate in a programme, it does necessitate that MYD has very clear, tight, and enforced criteria around participant targeting in order to maximise the impact of that funding.

When thinking about matters of participation, the question that we need to ask is 'who is likely to make a poor transition to adulthood without a formal youth development intervention'. In the case of this review specifically, the question is 'who is likely to make a poor transition to adulthood without the intense support provided by a structured youth development programme?'. MYD needs to be able to answer that question, and develop clear guidelines that help narrow the target group beyond the very broad Cabinet-mandated guidelines, if maximum value is to be obtained from the SFYP fund. Providers' experience should be used to help shape those guidelines but at the day-to-day level, their judgement should come in when determining whether individual young people fall within scope of these guidelines.

To answer this question, we need to consider who is conceptually most in need of assistance and who is most likely to benefit from a structured youth development programme. We also need to consider a number of other factors, such as:

• who the government/minister considers to be the target group
• who is legally able to participate
• who is practically available to participate
• the availability of other, more suitable pathways for young people
• safety
• group norms.

All of these factors need to be considered within the broader context of MYD's service and funding strategy, which would set out MYD's intentions with respect to matters like:

• what MYD wants to achieve through its funding of services (eg is it to trial and subsequently migrate successful programmes or elements thereof into the broader sector, as is stated in the SFYP cabinet guidelines)2
• the kinds of organisations MYD wants to develop through its funding (eg large organisations capable of delivering multiple programmes vs single operators who run a single programme)
• the areas where MYD wants to have services (eg does MYD want to provide services in areas where there currently are none or does it want to fund them in areas where there are a range of other services that can be drawn on either to provide additional support during a programme or post programme as ‘the next step’ for the young people)
• the kinds of activities MYD wants to be available to young people (eg does it want to fill gaps by funding, for example, arts- and drama-based programmes to ensure a range of activities exist for young people or does it want to be a specialist around a single or small number of programme types?).

As a first step towards developing guidelines, a mix of principles and more practical criteria are proposed that address which young people should be targeted for inclusion, which should be excluded, and which young people inclusion should be conditional for. These are set out in Appendix 4. In summary, it would seem the group most appropriately targeted for participation in programmes like NZCC and YSC are those young people who lack strong attachments to pro-social settings, who are disengaged or at risk of becoming disengaged from positive activities, and who need to develop foundational skills, attitudes, values and competencies in order to be able to successfully participate in educational or employment-related activities.

Referrals and Filtering

If the intention is to shift the focus of these programmes towards 'low-end at-risk', then referral mechanisms need to route in that portion of this group whose needs will be met by this type of programme. At the same time, filtering mechanisms need to identify and exclude those who fall outside these parameters and those whose needs are better met by some other form of intervention.

Police, Court and Family Group Conference referrals may no longer be appropriate referral sources, although as with most things, some discretion is always useful. More appropriate referral sources would include the young people themselves, families, schools and other agencies working with young people.

Developing a useful, formal filtering tool for providers to use at the initial intake stage is quite a complex task, although certainly not impossible. There are often matters of judgement that are needed when determining suitability; such as recognising when someone is ‘on the cusp’ between a negative and positive life trajectory. Filtering tools can lack the sensitivity to be useful in these circumstances, which may result in providers over-riding the tool’s findings anyway. Certainly there is an element of discretion that needs to be used when choosing participants, especially when it comes to considering the composition of the participant group as a whole that a formal filtering tool is unlikely to replicate. Even assuming providers were supportive, (not a given), it may be that the development of a filtering tool is relatively low priority relative to other action required. MYD's greater contribution may well be the development of more precise 'target participant' guidelines for providers to follow.

Programme Goals and Outcomes

Generic aims and outcomes for services and programmes funded through the SFYP fund were agreed by Cabinet. The key aim of MYD’s programmes and services is to enable the implementation of the YDSA, promote effective cross-sectoral responses for young people and facilitate the recognition, development and implementation at a regional and local level. The key outcomes of SFYP services are: improved self esteem; personal development and learning opportunities; improved connection to family or community; improved health; reduced re-offending (where applicable); and/or entry into further education, training or employment following completion of projects.³

There is a single set of aims covering both NZCC and YSC, articulated in the current service contracts as follows:

• deliver a youth development curriculum for young people in order to achieve positive outcomes
• deliver services that help young people acquire new skills, increase their confidence and motivation, support their active participation and remove barriers to education, employment and training
• deliver services which facilitate participation, inclusion and engagement, leading to achievement, awards or qualifications, long term economic and independent wellbeing
• deliver services which respond to current and emerging issues locally, regionally and nationally.

The outcomes, listed in the introduction to this report, are the same for both NZCC and YSC. These relate to the four broad areas of ETE participation, health and wellbeing, personal development and social development.

Overall, the stated aims and outcomes of both the SFYP fund and the NZCC and YSC programmes are consistent with the youth development literature. On a minor note, the SFYP fund outcome of 'personal development and learning opportunities’ involves activity rather than outcome. The more substantive issue is that both the SFYP fund and the NZCC/YSC aims and outcome statements are very broad. With respect to the SFYP fund, this breadth is arguably appropriate. In respect to NZCC and YSC, it is possible this breadth is counter-productive.

On the 'plus side', this breadth does mean that all areas MYD may wish to effect change or growth in through NZCC and YSC are captured. The possibly limiting effect of this breadth is that it encourages programmes to ‘do everything’ with young people. The risk, when this happens, is that very little is achieved because efforts are too dispersed and lack the intensity and/or the focus needed to facilitate specific outcomes.

The literature provides strong evidence that a twenty week programme cannot undo all of the harm that some young people have already experienced, nor can it accomplish what multiple institutions with infinitely greater resources have failed to do in the decade or more prior.4 Given this reality, it is vital that the programmes' aims/goals and associated outcomes give priority to those areas where action/change is most needed and is realistically achievable given the young people, the resources involved and our knowledge of the kinds of changes that programmes to-date generally have and have not been able to achieve.

This review contends that a youth development programme of this type, targeting this kind of young person, can most usefully contribute to young people's lives in the following way:

• by helping participants aspire to a life that includes positive and full economic and social participation
• by helping participants identify what their particular path to positive and full economic and social participation may look like, and the steps towards those goals
• by helping participants form enduring connections with positive people and settings that will help them to achieve positive and full economic and social participation beyond the duration of the programme
• increasing young people's motivation, confidence and self efficacy sufficiently in order for them to carry through on their next steps
• by helping to instil the knowledge and basic practices necessary for them to successfully carry out their next steps.

 While the existing set of programme aims and outcomes does in essence capture the basic logic set out in the above points, they do not do so as explicitly. The risk is that programme providers, who will invariably make decisions about programme activities based on their perceptions of what is most important, may emphasise a particular combination of aims and outcomes that do not reflect the logic outlined above.

To illustrate this point through example: an individual provider or youth worker may reasonably respond to the breadth of aim and outcome statements by choosing to prioritise a portion of them; those that seems most consistent with their beliefs and knowledge of what makes a difference in the lives of young people. An individual youth worker may decide, for example, to place their greatest emphasis on carrying out activities that increase motivation, self efficacy and so forth, reflecting a personal belief that these are young people’s primary barriers to participation. The literature, however, tells us that programme gains in this area are typically short lived; the extent to which the youth worker is aware of this is likely to be highly variable, and, at worst, quite limited, especially if their main source of knowledge comes from observing young people increase in their confidence and motivation during a programme. Unless the programme specifically includes substantial and meaningful goal exploration and personal planning activities, and deliberate equips young people with specific information and access to resources to follow this plan through, a worst scenario could involve young people ‘feeling better’ for a short period of time post programme and then returning to inactivity.

It would be useful for MYD to articulate a much clearer view of the basic logic that they perceive underpins NZCC/YSC programmes, whether it is similar to that discussed above or reflects a different perspective. This information needs to be communicated to providers, who need to give priority to the outcomes and activities that would appear to best support it. Further thought would be needed as to whether any possible re-articulation of logic/focus could be accomplished within the scope of existing contracts.

Another area for consideration involves the specific outcome area of post programme ETE activity. Current contract specifications require that 70% of young people should move into ETE post programme. This emphasis on ETE-type outcomes is a common practice in youth programmes internationally. At least in part, this reflects our understanding that continuous engagement and activation in useful and appropriate activities and settings is key to facilitating young people’s transition into productive adult life.

For many young people, the next step from a youth development programme will be an ETE activity. There is a portion of young people, however, who may need other steps before they are ready for ETE (eg some kind of therapeutic intervention). Placing such a strong emphasis on post programme ETE participation has the potential to effectively penalise those providers who accept young people on to their courses, knowingly or otherwise, whose next step is not an ETE one. To look at this practically: on a course of ten people, if two need other intervention before ETE participation is realistic, that means a provider can only have one young person decide not to move into ETE activity post programme completion before they fail to meet their contractual obligations.

The extent to which a 70% ETE target is realistic depends in part on the profile of the young people participating in the programmes. In recent times, MYD structured youth development programmes have been targeting 'the hard end', who are more likely to have additional issues that need addressing post programme than the lower risk young people that are now being proposed as the target group. With respect to participant group at least, the overall ETE target seems more achievable now than previously.

Programme outcome attainment also depends in part on the emphasis given to personal planning within a programme. It is highly likely that young people who leave a course without having explored their career aspirations, options and next steps, will struggle in a way that those who have considered these matters may not.

The World Bank notes that the ability of programmes to assist young people into employment is also closely tied to the prevailing macroeconomic conditions.⁵ At the current time, the labour market is contracting which will make it more difficult to place young people generally into employment. In some NZ locations, it may be almost impossible to do so. The ETE targets encourage participation in training and education as well as employment, which to some extent buffers providers in respect to outcome attainment. At the time of completing this report, however, it was becoming apparent that training and education providers are struggling to meet demand for places and vocational options for some young people are becoming extremely limited.

What these factors highlight is the challenge of setting realistic targets that span multiple years and potentially different economic and political climates. Most of the current service contracts of NZCC and YSC are locked in for a three year period. Making adjustments to these targets is not straightforward, although it certainly is possible, assuming there was a more appropriate target. At the same, it may be useful to consider what a suitable response would be should it appear these targets have become unrealistic.

Programme Structure and Content

Overall Programme Construction

The literature indicates there is value in providing intensive, full time, structured, multi-strand youth development programmes aimed at broad-scale, often quite foundational, development of young people. Reports from a range of sources also support the impression that there is a need for these programmes. At the same time it would seem that, in their current form, the programmes are unlikely to meet the Minister’s expectations regarding coverage.

There are three dimensions that affect coverage that can realistically be changed while holding funding constant. They are duration (currently 20 weeks), intensity (ie 'dosage', currently full- or near- fulltime) and volume (currently 8 -12 young people per course). Financial-related dimensions have been excluded because increased funding seems unlikely in the current fiscal environment and it seems reasonable to assume that providers are not able to do more for the same amount of money, or at least not in a sustainable way. It is also possible to adjust different combinations of these three dimensions in varying ways.

There is an inherent tension between attempting to increase coverage through some form of scope reduction and the tendency I suspect exists for programmes to sometimes try to be 'everything for everyone'.⁶ In part this tendency reflects the fact that many of those working in this field are extremely and passionately committed to young people and possess a very strong desire to do whatever is needed to help these young people on their path. It is also a reasonable response to the reality of service levels in parts of the country where there really may be no other locally-based programmes or services to refer young people to.

As discussed previously, there seems to be very little empirical evidence to guide decisions about duration, intensity and group size for the programmes like NZCC and YSC targeting this broad age range. The most useful approach may well be to present the scenario of greater coverage requirements to providers and see what they suggest. Their advice can then help to shape a modified structured youth development programme, in conjunction with the insights this report provides about important components and features of effective youth development programmes.

When having this discussion, one of the issues that should be foremost in people's minds is the importance of creating stability/continuity of involvement by useful adults in young people's lives. As has been explored previously, a young person that has an ongoing relationship with a positive adult who can guide, encourage and support them is likely to do better than someone who doesn’t have this. This basic belief provides part of the underlying rationale for youth development programmes generally and for the specific focus on helping young people to build positive connections in their natural settings.

Where positive connections are in place, the young person should no longer need the intensive assistance provided through a youth development programme or youth worker. Where such connections are not attained, however, there is considerable evidence to suggest that the ongoing involvement of a professional (in this scenario, most likely a youth worker) would be beneficial. A seemingly common tendency, however, is to use this scenario as a basis for justifying longer duration youth development programmes – the argument being: change happens where a young person trusts the youth worker; trust takes a long time to occur as does the resultant positive change; therefore the programme needs to be long enough to allow this change to happen.

What is in fact necessary is that the young person maintains access to a trusted and useful adult over time; it is not necessary that this is achieved through a fulltime, intensive (and expensive) structured youth development programme. One response may be to create the capacity for an ongoing mentoring strand of activity beyond the duration of the fulltime group component of the programme, for that sub-group of young person who need it. We know that some organisations and youth workers already devote considerable time and energy assisting young people who have left their programmes for this reason. Formalising this activity would be a way of ensuring it occurs in a more systematic, reliable and sustainable manner.

MYD's Contribution to the Choice of Programme Activities

The literature, at least at this point in time, suggests there is no single type of activity that can be considered particularly valuable or more 'youth development-ish' in nature than others; rather they are more or less relevant depending on interest and need. This supports the notion that providers decide which activities to include for any given programme.

For this approach to work well, it is essential that providers make informed choices about programme activities. This means that providers need to have a very clear understanding of the characteristics of different activities, or the kinds of opportunities they provide, and the links that exist between those opportunities and the desired developmental outcomes. Without this knowledge, activity choice may reflect what providers know or young people like, as opposed to what is most like to produce the desired results. In this scenario, the successful attainment of programme outcomes is more a matter of chance than design.

The impression formed during the review is that this kind of understanding does not yet exist throughout the sector. MYD has a significant role to play in this educative process, both in terms of generating the information and in its communication.

Having acknowledged that providers need to be able to adapt their choices of activity for different groups of young people, MYD’s decision to set no core components for the programmes may still be questioned. Previously, the service contract required that all programmes contain service activities, challenging recreation, education, work experience, and te Ao Maori. This approach ensured that providers carried out a useful range of activities while still leaving considerable flexibility around the specific activities and projects included within individual programmes. 'Challenging recreation', for example, could involve any number of different activities based around sport, outdoor activity, arts, drama, heritage and so forth.

For such a 'hands off' approach to work well, there would need to be a high level of capability across the sector, especially in terms of up-to-date ‘best practice’ knowledge. The extent to which this is currently the case may well be questioned. Providers need flexibility in order to achieve outcomes; the ‘managing for outcomes’ approach is premised on this notion. Even so, broad parameters that provide some direction for providers’ effort may be very useful, even as a time-limited measure, helping to promote greater consistency in programme quality.

Choice of Activities

If we know that activities are more or less relevant, or potentially interchangeable, depending on interest and need, then the evidence about the opportunities provided by individual activities and their apparent contributions to outcomes becomes even more important. This section considers the potential value and relevance of each of the traditional core components of NZCC and YSC in light of the literature.

No compelling reasons were identified in the literature for favouring conservation-based activities over other types of activity. Arguments are made for bringing young people in contact with nature and the outdoors in order to counter a possible ‘nature-deficit disorder’,⁷ and without doubt ‘the outdoors’ provides an excellent setting for developmentally-focused activities. No evidence was identified to suggest, however, that this type of activity may have a greater impact on developmental outcomes than other types. Ultimately, the deciding factor may be the extent to which young people are attracted to the outdoor setting as opposed to other developmental mediums such as drama and arts.

MYD currently allocates the bulk of its funding to NZCC programmes. Recognising that activities are the way in which programmes attract participants, in all likelihood, this emphasis on conservation means a group of young people MYD would consider suitable/in need miss out on an MYD-funded structured youth development programme because they do not like those kinds of activities. The extent to which this is an issue depends both on MYD’s desire to fund programmes of relevance to a broad group of young people through that portion of the SFYP fund and the availability of alternately-funded programmes involving other types of activities.

This argument also applies in respect to a 'challenging recreation' component. Young people who dislike physical activity and have participated in programmes that include this as a component have described the experience as demoralising.⁸ In theory, 'challenging recreation' could involve a range of activities that focused on, for example, drama, dance or art, as well as the more obvious ones like outdoor adventure. According to MYD staff, however, this latter type of activity tends to dominate. It may be that this is what providers know is most relevant for young people in their area; it could equally be that this is what providers are organised and equipped to do.

The only literature located that particularly endorses outdoor adventure activities over some other types was the work of Hattie et al.⁹ Their meta-analysis of adventure programmes suggested that, while the outcomes attained were more or less consistent with other education-focused programmes, the continued gains and longevity of those gains may be greater. The overall picture of efficacy depicted by the study was, however, quite mixed, and it does not suggest that this form of activity should be favoured above others. If it wished, MYD could encourage a greater range of 'challenging recreation' activities.

The evidence provides support for including a service component in the programmes. Done well, service activities provide a broad range of useful opportunities, including the opportunity for young people to positively and meaningfully interact with people outside the programme, which are in turn linked to a number of different important developmental outcomes. At the same time, local communities benefit from the completed project. Done poorly, such as when the nature of the project isolates young people from the community or the work involved is uninteresting or unlikely to develop useful skills, it is difficult to see benefit in it. There is substantial value in MYD carefully assessing the appropriateness of projects proposed by different providers as part of the overall monitoring process.

Education is a broad class of activity and one which provides within it considerable scope for adapting course content to reflect participant need. The literature provides strong support for the provision of information and teaching of skills across a broad range of life areas. The World Bank, for example, concludes life skills training positively impacts on the employability and educational outcomes of at risk youth as well as reduces specific anti-social behaviours.¹⁰

The value or appropriateness of teaching literacy and numeracy within youth development programmes of this type is less clear. It is quite probable that a sizeable portion of the young people who participate in a youth development programme have inadequate literacy and numeracy skills. Equipping young people with these skills is both important and necessary; however, it can easily be argued this is the role of educational specialists not youth workers. If programmes are to be shortened, for example, it makes more sense to focus on building confidence, motivation and so forth so the young person is better placed to move into a literacy and numeracy-focused course post programme.

Work experience is a popular component of youth programmes, assumed to increase employability and thus the move into employment, either within the organisation the work experience occurs in or an alternative site. The youth development literature does not clearly demonstrate how or to what extent work experience within youth development programmes affects participant outcomes. An MYD-conducted review of NZCC and YSC concluded there was a "clear correlation between work placements and ETE outcomes";¹¹ the methodological basis for this conclusion, however, seems quite weak. Work experience programmes falling within the ETE category are much more carefully evaluated; even so, the findings are mixed.

It is entirely plausible that exposure to work settings within youth development programmes will help young people develop important competencies. As with other activities types though, the extent to which gains are realised will likely rely on the careful selection of suitable work sites, the provision of support throughout the placement, and deliberate activities to support reflection and learning post-placement. Again, careful examination of work experience activities by MYD staff may be useful here.

The inclusion of a cultural component within a youth development programme is endorsed through the literature, in the sense that a clear sense of cultural identity and the capacity to operate in multiple cultural contexts are considered key assets needed for successful adulthood.¹² This does not mean ensuring a programme is relevant to the setting it is in or providing a 'Maori component' for Maori participants, although both are important. It means deliberate action to help individual young people explore and ultimately develop their own cultural identities as well as equipping them with the ability to successfully navigate multiple cultural settings. In NZ, this specifically includes an understanding of things Maori. While now quite dated, KPMG's 1990 evaluation of NZCC found the programme had been instrumental in teaching participants about "Maori perspectives" encompassing topics like the Treaty, Maori tradition, "my whakapapa" and so forth.¹³

In terms of overall balance of effort, an argument could be made for a strong emphasis to be placed on building positive connections within young people's natural environments (or helping them to access positive environments), as opposed to prioritising, for example, skills development. The design of the review prevents informed comment on the extent to which providers currently focus on connection-building as programme outcomes. The very limited impression gained is that the extent to which this occurs, and is done well, is probably quite variable.

Assessment, Goal Setting and Personal Planning

Assessment

As part of the current registration process, providers are required to ensure participants complete a nine page MYD-produced document to be returned to MYD within two weeks of programme commencement. The first three pages cover relatively standard enrolment information. The fourth, fifth and sixth pages of the document ask a series of questions that might possibly, once completed, represent the needs assessment listed as a required input in the service contract. I have been unable to establish if this is actually the case; no other MYD-provided documents have been identified that might constitute a needs assessment and MYD has not, at least recently, expressed an expectation that organisations have their own needs assessment approaches or tools.

The remaining three pages of the nine page document involve the creation of a personal plan. Two of the pages young people are expected to complete involve a multi-step goal setting process relating primarily to the period covered by the programme but also, to a limited extent, to the post programme period. These goals can potentially span any aspect of a young person's life. The goal setting process is tied in to the youth development programme by a single question which asks how the programme can help to achieve the goal(s) set. The development and implementation of this personal plan, along with subsequent monitoring of the plan, is also listed as a compulsory input in the service contract.

There are a number of issues that could usefully be (re)considered with respect to the approach to assessment and personal planning that MYD currently encourages through its nine page 'enrolment' tool.

The approach to assessment encouraged by this tool, if indeed the tool does aim to support needs assessment, seems fairly questionable in terms of its appropriateness and its value. Part of the reason for carrying out assessment within a youth development programme is to learn more about the young person and their life in order to establish how best to work with them, as well as to identify any issues that may impact on the young person's ability to safely and meaningfully participate in the programme. A provider could help a young person complete this form and still know nothing about issues or conditions which could impact on course participation, such as any recent or significant mental health issues or unsafe living arrangements.

As is the case with the planning process discussed below, there is an issue about the willingness of young people to divulge personal information in early stages of a course. At the same time, it would seem prudent to at least attempt to obtain a minimum amount of information in the early stages rather than waiting until the young person hopefully volunteered it at a later stage. Tools like the internationally-recognised HEADSS measure¹⁴ suggest useful areas to explore include the home, education/employment, activities, drugs and alcohol, sexuality, suicide risk and safety. According to HEADSS guidelines, each of these domains needs to be explored in relation to safety, connectedness and actual behaviour.¹⁵

The value or necessity of including a standardised assessment tool as part of the assessment process, within this particular context, is more difficult to establish. Ultimately, arguments can be made both for and against it. On the 'for side', courses like NZCC and YSC provide access to young people who may otherwise be disengaged from important core services. The systematic, as opposed to ad hoc, exploration of areas like psychosocial and physical health and wellbeing for individual programme participants may be a useful contribution for NZCC/YSC to make, especially in light of the emphasis placed on health and wellbeing in the current service contract.

Advocates of formalised assessment may argue that failing to incorporate a standardised assessment such as HEADSS within programmes like NZCC/YSC represents a wasted opportunity. Requiring providers to use a formal assessment is a tangible action MYD can take to encourage providers/youth workers to systematically identify important areas of need in the lives of participants.

At the same time, the successful application of any assessment tool relies on the involved staff possessing enough knowledge not only to administer the tool but to interpret results. This requires knowledge of the broad area of adolescent development, health and wellbeing, as well as tool-specific information; both of which have training and resource implications for providers, and in turn, for programme funders.

Critics further point out that identifying need is very different to meeting it.¹⁶ An assessment process is only valuable if the findings are acted on appropriately. To do this, staff need to know which services to direct young people to and how to locate them, which is in turn dependent on those services being available or accessible. This applies, however, regardless of the formality of assessment process or tools used.

Goal Setting and Personal Planning

Personal planning, particularly with respect to life post course, is a critical element in a youth development programme as conceived of in this review. At the current time, there seems little that directly encourages providers to undertake a comprehensive planning process as part of the programme. While providers are expected to achieve a target of 70% ETE participation post programme, this can be achieved with the aid of a sympathetic labour market or good contacts with training organisations and in the total absence of any comprehensive or meaningful longer term planning. The personal planning template provided by MYD, if anything, serves to downplay planning’s significance. While there may well be some providers who choose to incorporate comprehensive planning in their programmes, the current planning template may encourage a 'complete it, return it, tick the box' kind of mentality in others.

Ultimately, successful youth development means that young people are engaged in positive settings and activities over the long term, not just the short term, and programmes have a contribution to make beyond simply an initial post programme placement. Requiring providers to undertake comprehensive personal planning, where young people are encouraged to think about their longer term aspirations and how to achieve them, would seem a useful step for MYD to take.

In respect to the goal setting process set out in the current enrolment form, there are a number of specific issues that need to be considered. Firstly, the very detailed, multi-step method current used is likely to be unacceptably challenging for some young people. Secondly, young people may have little or no sense of their aspirations in the very early stages of the programme, when the form is to be completed. Thirdly, young people may not be willing to articulate their aspirations so early on, when the rapport and trust between programme staff and the group has yet to be established. Brief discussions with providers on this topic suggest all of these issues currently apply. Pressing young people to complete the current planning tool within the first two weeks seems quite likely to be counter-productive, serving to create or further reinforce a sense of incompetence or inadequacy and discomfort.

If MYD agrees with the value placed on personal planning suggested here, it would be worthwhile considering how best to ensure programmes consistently undertake comprehensive and high quality personal planning activities. Some time spent with providers developing a modified approach that providers consider more suitable would seem a useful action for MYD staff to undertake. A more simplified goal setting method, occurring later in the programme, may be useful. As part of the data/information work stream, MYD should also consider what assessment and planning information it needs to receive: currently, MYD receives (at least in theory) planning and goal setting information for each programme participant. It is unclear what this information is intended to be used for and the usability of this information in its current form seems low.

Performance Measurement, Monitoring and Evaluation

Work is currently underway to improve the broad activities of performance measurement and monitoring for the NZCC and YSC programmes. This includes the development of more relevant data capture and reporting tools. This work needs to be underpinned by a clear understanding of the information needs of different stakeholders. It also needs to be based on a clear understanding of youth development principles and what is important in youth development practice.

Overall Information Framework

MYD needs an overall framework to guide its information activities (performance management, monitoring, and evaluation) which directly focuses on the various elements the literature suggests are necessary for good youth development practice. This framework should be based on a clear description of programme theory, including the theory of change that underpins the structured youth development programmes. Providing clear descriptions of the rationale for the inclusion of different activities in the programmes, alongside statements about the opportunities the activities are expected to provide and their expected links with programme outcomes, will also usefully inform the framework.

Inputs

Working through the different components of a programme, beginning with programme inputs, MYD needs a fuller picture of programme participants. The kinds of areas that should be covered include participant demographic and characteristics information, as well as strengths and needs information. Developing a more suitable tool for use by all providers will be a useful action for MYD to undertake. Using internationally recognised measures or categorisations wherever possible should help with the overall comparability of information with other programmes.¹⁷

Some thought needs to be given to the amount or type of information MYD needs about programme inputs like materials, equipment and facilities. A key question to consider is how important or useful is it to build a picture of this aspect of programme provision, relative to other information needs.

The possibility of more systematically capturing information about programme instructors/youth workers should also be considered. The skill of a youth worker is considered one of the most influential factors affecting participant outcome, yet MYD does not yet have a comprehensive picture of the people delivering its programmes. This information would also be useful when making decisions about the need for sector development, as well as in making service comparisons across providers.

Activities

A much clearer picture of programme activities is needed to round out the overall picture of programme performance. It is difficult to make judgements about a programme’s value, particularly in terms of its contributions to participant outcomes, with limited knowledge of what activities actually occurred or the quality of those activities.

Currently providers have complete autonomy in deciding programme activities (ie work experience vs recreation vs life skills training etc), which means there will be considerable variation between the different programmes even within NZCC programmes or YSC programmes. Regardless of whether or not MYD elects to provide greater direction on the types or broad classes of activities that will be used, there is value in trying to obtain more systematic information about what providers actually do during the courses. While some information is currently collected on activities, further attention to the form in which it is collected would help to improve its usability. It would be useful for Ministry of Youth Development to develop and require providers to report against a limited number of activity categories, such as 'personal planning', 'one-on-one time between young person and youth worker', 'work trial' etc.

It would be useful for Ministry of Youth Development to examine activity and financial records to establish the extent to which providers are contracting out responsibility for delivering different activities. Some activities, such as literacy and numeracy instruction, involve specialist skills that youth workers would not be expected to have. Contracting out this work to an expert is a sensible response to this situation. Other activities, such as the conduct of challenging recreational activities, fall within the sphere of expected skills for a youth worker in many of the conservation or outdoor-focused programmes. Contracting these activities to external individuals or organisations raises serious questions around the capability of these organisations to conduct aspects of their core business.

Information is also needed to enable MYD to drill down into the 'features of settings' that the literature identifies as being critical in effective youth development programmes. Pre-course discussions with staff, examination of course schedules (including intended projects), and site visits during the course will provide information about the features that providers intend and actually do incorporate into individual programmes (eg safety, appropriate structure, supportive relationships, opportunities to belong, to experience mattering and so forth).

These methods could also be used to examine the aspect of programme activity termed 'therapeutic approach' in this report. Youth workers who are unable to describe, if not some underpinning theory then at least their practical approach to engagement and learning, are unlikely to be carrying out activities in a way that will facilitate the kind of development MYD is seeking to facilitate through the programmes.

The conduct of goal setting and personal planning, if MYD decides to place greater emphasis on this area, should also be examined through a monitoring process. In the first instance, there is a compliance aspect which needs to be addressed: these activities represent a specific deliverable within the current service contract and MYD needs to know they have been carried out. At the same time, the quality dimension also needs to be assessed. Again, this could include an examination of providers' general processes and tools, alongside a real-time or retrospective assessment of the quality of outputs through an examination of a sample of assessment findings and plans. If MYD elects to place greater emphasis on building participant's enduring connections in their own environments, then carrying out activities like social mapping will be useful within programmes. As for the other activities discussed here, there are both compliance and quality dimensions to this work.

Outputs

For each course conducted by providers, MYD needs to know how many young people completed the course as well as how many exited early. Capturing the reasons young people exited early, and where possible what they did on exit, is an important part of this picture. Understanding the reasons why young people chose to remain on the courses is also important.

For each course, it is important to establish 'what' and 'how much' service participants actually received. Without this information, it will not be possible to determine, in the event an appropriately-designed evaluation is carried out, which components of a programme 'made the difference'. Counting this type of output will be easier once MYD has developed an approach for classifying and capturing programme activity information. It will also be complicated, however, where providers hold an integrated contract and can move young people between different programmes at will.

Outcomes

The current basis for assessing change/achievement in participants seems insufficiently rigorous to meet the kinds of information needs stakeholders have indicated they have with respect to programme outcomes. Further work is required to clarify or refine the outcomes Ministry of Youth Development expects will result from participation. Once this is done, more relevant and precise outcome measures need to be developed. Having pre- and post-measures for assessing progress would be of considerable value, while standardised measures would help to make any evaluation work ultimately carried out a more valuable contribution to the overall body of evidence on youth development programmes.

Impact

The only impact evaluation that has been conducted is de Boer and Soughtton’s work looking at MSD’s employment assistance programmes, within which they include NZCC.¹⁸ The issue with this work is that it treated NZCC as a work confidence programme, which it is not. Unsurprisingly the evaluation found that NZCC is a highly ineffective work confidence programme.

The challenge MYD now faces is that this study has the most technically rigorous methodologies of those evaluations that have been conducted on NZCC/YSC. In the absence of an equally rigorous evaluation where NZCC/YSC is assessed as a youth development programme, these findings will continue to attract attention regardless of efforts made to explain why this evaluation provides a very incomplete picture of potential programme accomplishments.

If MYD is to move beyond this situation, a new impact evaluation is required. This type of evaluation is complex, time consuming and resource intensive. Accordingly, it should only be conducted once programmes are sufficiently developed, and stable, to make it worthwhile.

Footnotes

¹ Note that current service contracts specify the target group as youth aged between 15 and 24 years of age who are vulnerable and disengaged, with those aged 15 requiring a formal school exemption

² SDC (05) 19 refers

³ Ibid

⁴ Higgins, 2003, op cit

⁵ Cunningham et al, 2008, op cit

⁶ Note that 'scope reduction' could also involve a perceived 'dilution' of accomplishments stemming from increased course sizes, as well as the more obvious reductions in duration or intensity/dosage

⁷ Louv, 2006, op cit

⁸ Weaver, S (2001) cited in Barwick, 2004, op cit

⁹ Hattie et al, 1997, op cit

¹⁰ Cunningham et al, 2008, op cit

¹¹ Ministry of Youth Development (1995), Overview of the internal review of the New Zealand Conservation Corps and the Youth Service Corps. Wellington, NZ: Ministry of Youth Development, p5

¹² Eccles and Gootman, 2002, op cit

¹³ KPMG Peat Marwick, 1990, op cit, p8

¹⁴ HEADSS is a screening tool for conducting a comprehensive psychosocial history and health risk assessment with young people. For further information, see, for example, clinical guidelines at www.starship.org.nz

¹⁵ www.starship.org.nz, retrieved April, 2009

¹⁶ Gilligan, P. and Manby, M (2008), The common assessment framework: Does the reality match the rhetoric? Child & Family Social Work, 13 (2), p177-187

¹⁷ See, for example, the compendium of research and assessment tools for youth development: www.rmcdenver.com/cart; and the National Clearinghouse for Families and Youth, www.ncfy.com/publications/satools/index.htm

¹⁸ De Boer, M. and Soughtton, D (2008), Sustainable Employment Annual Report 2008, Annex 1: Evidence by programme type. Wellington: Centre for Social Research and Evaluation, MSD