A working document from Kerry Shephard
1 Some questions to ponder (preferably in small groups)
Why might supervisors wish to evaluate the impact of their supervision activities?
In what ways is postgraduate supervision similar to, or different from, other forms of university teaching and, therefore, to what extent should evaluation processes (developed largely for undergraduate teaching) be considered appropriate?
Are there some fundamental educational values at stake?
What processes are workable and appropriate?
How does the higher education research-literature help us?
2 Some resources to help
University of Otago Guidelines for Evaluation of Teaching
Chapter 8 Peer Review pages 69 and 70
Use of peer review for evaluating supervision
Peer review is one way of evaluating your one-on-one teaching such as supervision of advanced clinical students or supervision of postgraduate research students. Both forms of teaching preclude the use of student questionnaires because numbers are usually too small and the students know that their privacy cannot be guaranteed. The students are likely to be concerned that their working relationship with you, their supervisor, may be in jeopardy if they respond as frankly as you might wish.
Peer review can provide a means of evaluating one-on-one teaching and you are encouraged to consider it as a solution to a difficult issue. It is important that you do not miss any opportunity to include student supervision in your evidence of teaching performance for your Otago Teaching Profile.
The processes described above lend themselves to gathering such evidence. Some specific strategies include the following:
a. Select a peer who can evaluate the currency and appropriateness of the disciplinary content of the supervision you provide, perhaps by examination of each student’s written material and the feedback you provide; by observing your discussion with the student(s); by discussing with you your aims and practices in supervision.
b. Select a person, probably not the same person as in a), who has special skills in facilitating discussion. Both you and your students must have confidence in this person but they could be from outside your discipline and might be closer to the students than yourself (for example a recently completed postgraduate student or a Medical Education Adviser if in the Faculty of Medicine). The “facilitator” would be asked to meet with your students individually or as a group and hold a structured discussion on the quality of your supervision. The facilitator would then prepare a report for you which is seen first by the students so they can be assured that the report does not contain material which may cause unintentional discomfort to them or to you.
c. Other reviewers might be needed for other special aspects of your supervision especially if you are responsible for a variety of student projects.
Each of your peers would sign the peer review form (see p73) which is submitted and any reports which are prepared can be held in the on-call documents, listed as such and be quoted and analysed in the self-evaluation statement in your Teaching Profile. Your HoD can also be encouraged to comment on his or her knowledge of the quality of your supervision in the HoD’s validation statement (if for a promotion application) or in the HoD’s report (if for confirmation), with or without access to the reports in the on-call documents according to your wishes.
It is important that you give attention to the de-briefing process described in 4 [see footnote below] because peer review is too valuable and too demanding to be used only for promotion or confirmation procedures. Much can be learned from appropriate peer review.
Further guidance for peer review
To make the most out of peer review, it is suggested that collaborative partners consider seeking guidance. Please consult with HEDC.
[4. The debriefing session
Debriefing is a dialogue about teaching. It may take place immediately following the review session, but it can happen at a later stage or take the form of an ongoing dialogue. This is an extremely important part of the process but can also be difficult to deal with. It is usual to structure this session to the framework agreed in the briefing session. You or your reviewer may have observations which fall outside the agreed framework and it is your decision as to whether you wish to discuss these further. The debriefing session is not the place to bring up the possibility of extending the original agreement, although additional ideas will be generated during discussions and these can form an important part of professional learning. A peer reviewer is not there to tell you how to teach, or how they teach, but rather to explore teaching issues with you in the form of a mutual enquiry.]
3 A description of the Otago process developed by Professor Chris Heath (2004)
Case Study of Peer Evaluation Research Supervision at Otago University
What is it?
Peer evaluation (or review) is a term used to describe a wide range of evaluative practices characterised by being undertaken with colleagues. It can provide evidence of both your performance as a supervisor and valuable information for your professional development.
What are the key issues for peer evaluation?
The approach described here is that used at the University of Otago for both undergraduate teaching (usually to supplement student opinion surveys or to diagnose the cause of problems) and for postgraduate research student supervision (where the concerns are the small numbers of students and exploring issues which cannot be evaluated effectively by student questionnaire). The account below draws on the University of Otago Guidelines for the Evaluation of Teaching 2005.
Academic staff at Otago are encouraged to use peer evaluation for their supervision activities as a component of the evidence they submit for their teaching performance appraisals but there is also a strong emphasis on the developmental opportunities - the processes of evaluation are too important to be used only for summative purposes or only for formative purposes. There are three key principles in the Otago peer evaluation process. They will strengthen the experience for all parties if observed.
That it is voluntary;
That it is collaborative;
That it is done for the purposes of professional learning.
Academic staff are not required by the University of Otago to undergo peer evaluation, or engage in processes that involve one peer making summative judgements about one another. Peer review involves collaborative partners working together to learn about and improve their teaching practice. Working together in this way has the potential to offer critical insights into our teaching that cannot be obtained through other sources, such as student and self-evaluations. It should not be seen, however, as a substitute for other forms of evaluation.
Suggestions for conducting a peer evaluation
The first and perhaps most important decision you must make in undertaking peer evaluation is the choice of your reviewer. If the issues you wish to review have a disciplinary focus, you may need to collaborate with a reviewer from your own department. However, if you are dealing with more general issues of teaching and learning as they apply to supervision, then a peer from any discipline can be chosen. It is essential that the relationship established with your peers is built on mutual trust and respect, as the process of exposing our teaching/supervision to the criticism of others can be threatening. If you cannot establish such a relationship with your peer, then it is likely that the review will result in a defensiveness which is inimical to learning. On the other hand, it is important that the chosen peer is prepared to be critical and challenging. If the peer evaluation exercise is to be used for performance appraisal your peers must be able to establish that they have used a credible process and have behaved in a professional manner appropriate to members of the academic community whether or not they have a reputation as successful researchers.
An example of a peer evaluation plan which might result from considering the issues above could be:
Select a peer who can evaluate the currency and appropriateness of the disciplinary content of the supervision you provide, perhaps by examination of each student's written material and the feedback you provide; by observing your discussion with the student(s) in routine meetings or at special review times; by discussing with you your aims and practices in supervision.
Select a person, probably not the same person as in a), who has special skills in facilitating discussion. Both you and your students must have confidence in this person but they could be from outside your discipline and might be closer to the students than yourself, for example a recently completed postgraduate student or an educational expert. The "facilitator" would be asked to meet with your students individually or as a group and hold a structured discussion on the quality of your supervision. The facilitator would then prepare a report for you which is seen first by the students so they can be assured that the report does not contain material which may cause unintentional discomfort to them or to you. Your students will want to protect their working relationship with you at all costs usually in their own interests but sometimes out of exaggerated loyalty to you as their supervisor in a special part of their lives. There can be unexpected spin-offs from this form of focus group discussion inasmuch as your students may find it helpful - and enjoyable - sharing with each other their experiences working with you as their supervisor. The process is entirely compatible with evaluating supervision of students at a distance although organising the meeting is likely to be demanding.
Other reviewers might be needed for other special aspects of your supervision especially if you are responsible for a variety of student projects or there are special circumstances - there can be no standard formula for peer evaluation of supervision given that each situation will be unique.
There are a number of decisions which need to be negotiated before carrying out the review. Each partner will have to be clear about:
The aims and focus of the review process;
The way in which it will be conducted and the roles each partner will play.
In the briefing session with your peer evaluators it may be useful to begin by sharing some background details on the group of students being supervised such as the stage they are at (for example who is just beginning, who is writing up and who has completed and moved away), do your students work alone or in a laboratory team, are there any students being supervised at a distance and so on. As the person initiating the review process, you should then outline a proposed focus for the review. While identification of specific issues for review can be difficult it is preferable that a clear aim is expressed. It is equally important that the aim for a review does not become so ambitious that it cannot be met by the reviewers. This may require some reflection on what is important to your teaching practice. Other forms of evaluation or past experience may also suggest areas for exploration. All partners need to be satisfied with, and in agreement about, all aspects of the brief. Ultimately, if peer evaluation is to succeed, it will be necessary to adopt practices that suit both parties. It is especially important to meet with the focus group facilitator to decide on the core questions which will be put to the students particularly if you have some issues which really matter to you. Issues of confidentiality should always be clarified.
Whether or not you want to use the evaluation for summative purposes, a written statement from each of your peers is always valuable and will help each of you to complete the process you have agreed on at the start. At Otago, the actual reports cannot be submitted for performance appraisal, we are minimalist in the documents which can go forward to assessing panels, but a form is signed by each reviewer to establish that they were involved and the process which was used. The peer reports can be quoted in a statement on teaching which is submitted by the teacher and by the Head of Department and the reports are then held "on-call" in case verification of the way they have been cited is required. While indirect, the process does assist in achieving the delicate balance between using peer evaluation for summative purposes and for those valuable formative opportunities. It makes little difference; each report should be quite brief, professional in the criticism given, and constructive in its suggestions for improvement. As with conducting formal examinations of undergraduate students and writing references for job applicants the credibility of the peer evaluation report reflects back on the reputation of the writer so that bland commendations unsubstantiated by evidence do neither the reviewer or the reviewer any good and can also bring the process into disrepute. The crucial issue of the confidence of the students has been discussed above.
It is usually helpful to have a debriefing session with each of the peer evaluators, probably after they have given you their reports. If the process has been collaborative and in the spirit of mutual benefit there is much to be learned from talking with those who now know a great deal about your role in research student supervision.
4 Some research papers on this topic
Zuber-Skerritt, O and Roche, V (2004) A constructivist model for evaluating postgraduate supervision: a case study Quality Assurance in Education 12 (2) 82-93
Aspland T, Edwards H, O'Leary J and Ryan Y, (1999) Tracking New Directions in the Evaluation of Postgraduate Supervision Innovative Higher Education 24 (2) 127-147