Unskilled and Unaware of It

If you have never heard of the Dunning-Kruger effect then I highly recommend reading about it and hopefully reading the original paper (Kruger, J. & Dunning, D (1999). “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”Journal of Personality and Social Psychology 77 (6): 1121–34 – it is available to Salford staff via your Athens login.)

The Abstract reads:

“People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of the participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.”

What does this mean for teaching, learning and academic development? I have my own opinions but it seems to me that the key is development for all staff which is challenging because the problem is the less competent teacher does not know it and therefore will not feel the need for help. The more competent know they are but feel possibly less competent than they really are and so seek development to get even better.

So what do you think?

Advertisements

PGCAP Week 6 – Assessment & feedback

How do you reflect if you are not feeling reflective? I personally blame it on the clocks going back. November is my mardy month 😦 But anyway to answer the question, I think it is just dive in. Start writing, write anything and then let yourself fall into it and hopefully it should start to come. Also remember to start on a personal note and be conversational rather than formal. Which of course leads into one of our topics for today, feedback. I have learnt a great deal about feedback in the last 6 weeks. I think it is something we can all improve upon (giving and receiving feedback). It is such a personal aspect of human beings as social creatures and it is easy to forget there is a person on the other end of our feedback especially when it is in writing, which I don’t think is a natural way of giving feedback. This is why I am increasingly becoming a fan of audio feedback. I think it is great for formative feedback and I hope for our PGCAP participants that the message about feedback came across. I feel I know more about the theory of feedback than the practice of feedback so just delivering this session with Chrissi has made me learn more about it.

As those who attended might have guessed, assessment is something I am passionate about as I see it as the core of our courses and modules (although I am reminded of the reaction I got when learning BSL when I told my peers that I didn’t want to take the exam and therefore would not get the certificate, I just wanted to learn BSL, I didn’t want to sit an exam and the certificate meant nothing to me but a degree I guess needs the assessment to validate the students experience, skills & knowledge but please make the assessment about learning as well).

I found myself arguing strongly in favour of the employability and employer involvement agenda even though I am still suspicious and critical of this agenda. So, while I think we need to consider employability we need to be careful that we don’t turn our degrees in training courses and we keep the unique character of what a degree is about (critical thinking, academic rigour, development of the individual etc.)

I felt that assessment and all those bits linked to it is the single topic that really gets everyone going in HE. It seems such a contentious issue and everyone has an opinion. Is anyone happy in UK HE (or even in our whole education system) about what happens generally about assessment?

The session prompted some interesting debate which then led to trying to balance having the debate with what was planned. In retrospect, maybe we should have just ditched the plan and gone with the debate. Maybe a PBL type scenario for next semester or a proper organised debate with the content and / or questions made available online beforehand. Something to think about. Your thoughts welcome on this idea.

A slightly rambling post but isn’t that what blogging is about the process of thinking, learning, sharing and metacognition rather than a nice polished product.

First Level Assessment conference – part deux

As the taking the ‘ass’ out of assessment was such a good conference, I’m back with the lessons of the parallel sessions and final keynote.

What makes a good 1st year module? Anne Llewellyn

Anne talked about the usual issues for students in their first year such as expectations, self-belief, friendships and so on as well as the issues for staff mainly focused around the massification of HE. She used Ruth Pickford’s idea that 1st level assessment and feedback should focus on (taking a Jack & the Beanstalk reference) Fe Fi Fo Fun:

Photo taken by DaveC71

Jack & The Beanstalk (attribution: DaveC71)

  • FEelings – around issues such as improving self-belief
  • FIt – as in ‘for purpose’ so that they are aligned, empowering, interactive…
  • FOrmative – ie help students improve, summative assessment should be used sparingly.
  • FUN – in terms of engaging, motivating, authentic.

Some examples of good practice included having residentials, group work on authentic tasks, interaction. Reflection is critical for students to understand what is expected.

She also mentioned ways of reducing the assessment burden on staff and students. The example was using an interview to assess a portfolio. Get students to create a portfolio of evidence but rather than read through the portfolio and assess it the students have an interview which has to draw on the evidence in the portfolio. Very efficient for staff time. Also seems highly relevant for students as very akin to a job interview after completing an application form. In other words we develop and assess student skills at the same time as assessing their content knowledge.

Another key take home point was about immersion in learning and teaching and giving students the chance to experiment without it counting against them if it goes wrong.

Helping to students to be successful in assessments – Mantz Yorke

The guru of first year experience research gave the final keynote. He talked about five areas: language of assessment, expectations, standards, feedback & grades.

In HE we often talk about graduateness but what is this? Mantz partly defined it as moving from acquiescence to autonomy or from authority dependence to self-authorship.

acquiescence  →  autonomy

“What do you mean?”

Do we explain our regulations clearly to students? Are they sensible guidance or legal documents? Do we include examples to illustrate expectations in our assessments? Do we explain what the difference is between analysis and evaluation? The rules at university are different to school. I think one of the big ones is a deadline being a deadline. At school you tend to set interim deadlines because some won’t hand in on time and it gives everyone time to redo the assessment if it is not quite right.

Mantz made some interesting comment son standards. Saying that subject benchmarks are difficult to define and not written in the clearest manner. That Learning Outcomes are a proxy and inherently ‘fuzzy’. He then went on to look at the problem with criteria (e.g. assessment criteria etc.) Can we conceptualise criteria? What are the grade boundaries? (ie are they rather arbitrary?) How do we deal with multiple learning outcomes? Are some more important for others and therefore we get into weighting of outcomes and rules for combining bits.

What is feedback for?

We have evidence that formative feedback works (see Black & Wiliam (1998) Inside the Black Box). However, we don’t always do it well and maybe miss out aspects of formative feedback.

Typology of feedback

feedback table

We tend to focus on the top left (formal teacher feedback) and should move towards the bottom right.

Mantz talked about the quantity and timing of feedback and asked “Why write feedback if it is going to be binned by students?” Feedback should be present early on and tail off. In other words, “should we give feedback on summative assessment?” We should also encourage students to establish their own internal standards so they can accurately evaluate their own work. We should also look at creative ways of giving feedback. Such as peer feedback and giving choice over how students receive feedback.

What does a grade really say?

This is one of my favourite topics. Many universities and courses still cling onto percentage marking for assessment such as essays and the like where such fine distinctions are impossible to make. What is the difference between a 65% and a 67% essay? Do we really mark from 0 to 100% or is it actually 30-80%? We also assume that the distribution of such grades tends towards the middle in a bell curve, although with learning outcomes this tends to shift to the right. We know that different subjects vary greatly in their distribution of grades. Even different modules in the same subject can have wildly different grades. If you didn’t know already, when an 18 year old asks which subject gives them the best chance of a first, tell them to study maths. Although you might want to neglect to tell them maths also gives them the best chance of failing too. Why? because in maths they use the full spectrum of percentages.

Final thoughts

  • What about programme level assessment?
  • Being more intelligent about assessment, don’t assess a learning outcome more then once.
  • Make assessment relevant, e.g. by including contemporary events.
  • Learning leads to performance but do performance goals lead to learning? In other words, focus on the learning and a good grade will come but if you focus on getting a good grade you don’t always learn…and how do we convince the student of this?
  • Are students strategic or just tactical?

Panel Discussion

The panel discussion consisted of a number of notable national teaching fellows. Here are some of the quotes that caught my ear.

“Far too much written assessment, we are now writing for a degree not thinking / performing for a degree.” “What is in people’s heads is not always what they can put on paper.” Phil Race

“Students love audio feedback.” “Give group feedback on general features of all assignments.” Bob Rotherham.

“How often is too often and how much is enough [feedback].” “Feedback should be more frequent in the 1st year.” Sally Brown

“Staff don’t always know what students need in the 1st year.” “Our system does not recognise the support 1st years need.” “Our best teachers should teach 1st years.” Ruth Pickford

The Science of Learning webinar

You can tell we are now in conference season; I’ve got HE academy conference tweets coming in from colleagues at the conference in Hertfordshire, The OU annual Learning and Technology conference: Learning in an open world and have just finished an online webinar on the Science of Learning. Aaargh information overload.

This post is about the webinar hosted by the Association of Learning Technology in the UK. We were given two presentations:

  1. On some of the theories of learning by Richard Cox at Sussex University
  2. On the practicalities of learning by Donald Clark

The presentations and recording of the session can be found here.

Some of the take home points for me made by Richard were about metacognition, feedback and information presentation.

Richard talked about implicit and explicit learning. Our education systems try to make learning explicit and expects students to keep it explicit in the sense that they have to demonstrate it in assessment. However, research shows that ‘experts’ use implicit learning. This seems to me to chime well with notions of threshold learning. If you are an expert in something you can do it without thinking and therefore can find it difficult to make that implicit knowledge explicit for novices in a subject or skill. This was also echoed  in a practical sense by Donald who takes this further and says that if this is the case “Why is the majority of teaching done by people who know so little about the psychology of learning?” even though of course they are experts in their discipline.

Richard went on to talk about metacognition and how ‘good’ learners are self-aware. They know what they do know and don’t know. They know how to learn and how to think. That is not to say that feedback is not important. Feedback is crucial, particularly, one would assume, for those less aware learners.

We can present information in two modalities (I think there could be more but two were presented to us by Richard):

  • Linguistic, i.e. the written or spoken word – which remember is a construct of reality interpreted through our own thinking processes and nothing like actual reality (if such a thing exists)
  • Graphical – e.g. diagrams

We should mix and match these modes of presentation and not just focus on one or the other.

Donald was somewhat more controversial and I felt had a particular issue with certain ideas in training and teaching which meant that he tarred a lot of good practice as rubbish. Casting that aside he had a lot of useful things to say.

He opened the main content of his talk by criticising some ideas that are common in education as unscientific and in some cases blatantly bad for you!!! Now as a science graduate, I understand the scientific methodology and the need for good quality research, however a theory may be proven correct even though we don’t yet have the evidence to show this.  So Donald’s 10 pseudo-scientific myths: behaviourism, learning styles (see the Coffield report), left/right brain theories, Mozart effect, Piaget, Vygotksy, Maslow, NLP, Kirkpatrick, life coaching and similar.

On the other hand he gave us 10 scientific facts that often get ignored:

  • spaced practice – repetition of tasks overtime,
  • cognitive overload – our brains can not come with too much information in one go,
  • chunking – breaking information / skills into smaller pieces for easier absorption,
  • order – paying careful attention to the order of material (I think this also chimes with my pet theory of the week, Threshold Concepts)
  • episodic & semantic memory (didn’t get what he meant by this point, feel free to enlighten me!)
  • psychological attention – our brains just aren’t build to pay attention for long periods of time,
  • Context – this is massively important. You can not remove the content of learning from the context in which it is going to be applied.
  • Media Mix – cf. Richard’s earlier point
  • Learning by doing – speaks for itself really!
  • Peer groups – or at least getting the peer groups right rather than expecting collaborative learning just work on it’s own.

Donald asserted that too many courses are ‘padded’ by useless information and ‘gimmicks’ such as audio, images etc. Keep it focused on the core content, people!

Finally, he talked about how to increase retention of learning. Self-rehearsal, taking notes (in the learners own words), blogging (or I prefer reflective writing), repetition, delayed assessment (ie giving the learner chance to learn it properly not just from short term memory), recording (Donald is a big fan of recording lectures), spaced e-learning, mobile technology, games & simulations (these last two tie in very well by  learning by doing but in ‘safe’ environments.)

An interesting and thoughtful webinar. If they make the recording of it open access, I shall post the link here.

Inclusive Curriculum Design workshop

On Monday 21st June, Chrissi Nerantzi and I ran the first of our new academic practice workshops on inclusive curriculum design. The focus was on the principles and theory underpinning inclusive modules and curriculum. On of the key principles was that good practice already exists and being inclusive is about being aware of the good practice and making sure we do more of it. It is not about targeting different groups for specific help but about providing a variety of teaching, learning and assessment strategies. In the words of CAST when talking about the principles of Universal Design for Learning:

  • Multiple means of representation, to give diverse learners options for acquiring information and knowledge,
  • Multiple means of action and expression, to provide learners options for demonstrating what they know,
  • Multiple means of engagement, to tap into learners’ interests, offer appropriate challenges, and increase motivation.

Inclusive teaching is just good teaching!

Part of the approach to good teaching is to engage in dialogue with students. In order to meet the needs of students, we need to know what those needs are and not make assumptions. And students need to be clear about what we expect from them. So in our curriculum design we need to write clear aims and learning outcomes. We need to have clear assessment tasks that are explained well.

An extensive presentation is available here (http://prezi.com/sdhe0ofbc1p1/) with many useful references to further material. For staff at the University of Salford, we also have a Blackboard space for our Academic Practice workshops and if you would like to join please contact the LDU. For external visitors, we are planning to make our material open and available on our website later in the year.

The workshop went well. On reflection for next time maybe I should take my own advice above! Whilst I got the expectations from participants at the start what I didn’t make as clear was my own expectations. I think this is very important when working with students but also in staff training sessions as well. For me learning is challenging and the role of academic development is to bring about attitudinal and behavioural change. For anyone who has worked in equality and diversity training, you will know how difficult attitude change can be. Possibly the most challenging form of learning. So making that expectation clear is important.workshop

Another aim of the workshops is to model good practice in teaching and learning which I believe from the feedback sheets and my own reflections was achieved. One of the best aspects of the workshop was the dialogue and discussion between colleagues. Being able to share ideas and practice across school and faculty boundaries is immensely valuable. Hopefully later in the year we can develop a cross-school (college / faculty)  community of practice (see the work of Etienne Wenger) for those interested in learning, teaching and assessment.

Thank you for those colleagues who attended and for your feedback.

“Enjoyable, informative, good to work with colleagues from other schools.”
“Very, very interesting.”
“Very useful ideas, directions to further resources.”
“I plan to utilise all the helpful info learnt to improve my teaching.”
“Should be compulsory for senior management.

If you want to know more about curriculum design or inclusivity in the curriculum please contact the LDU. If you have any resources on these topics please leave us a comment.

Taking the ‘Ass’ out of Assessment, part 1

Context

I attended this excellent event at LeedsMet university on the 16th June 2010. This annual conference on First Level Assessment and Feedback is part of the dissemination strategy of LeedsMet’s First Level Assessment Project. It was so useful and there is so much to share I am going to break this into a number of different posts (congratulations to Professor Ruth Pickford and her team).

Introduction

– Professor Sally Brown (PVC at LeedsMet)

An interesting and engaging opening with Sally talking about her own experience of arriving at university for the first time. Although the traditional ‘away from home’, full-time, 18 year old student is no longer the norm, many of the key experiences are still important to consider today:

“I didn’t know the rules of the game”
“What is a seminar?”
“First in family to go to University”
“Homesick, lonely”

By remembering how it felt for us we can hopefully design activities that help students new to the university.

Opening Keynote

Professor Mark Schofield
(Academic Director of the SOLSTICE CeTL, Edge Hill University)

Success

Mark’s philosophy is one of collaborative learning through dialogue, “smart people in dialogue getting smarter”. For him the bottom line is it is a tragedy if students don’t succeed. Some of us may be familiar with the attitude of some in HE who say “it was OK for me so it should be OK for them (students)”. However, HE is no longer about sorting the ‘wheat from the chaff’ it is about helping students to succeed and giving them opportunities to demonstrate their knowledge and skills.

Assessment as Burden

Often staff and students can see assessment as a burden but surely “it ain’t worth doing if it is drudgery for assessors and students”. So we need to make assessment more efficient and effective. Let us not overburden students with assessment and what is the point of measuring the same learning outcomes multiple times? Assessment should be motivating for students.

Some theory and practice of assessment and feedback

“I taught X to dance and I assessed her too.” said Ted
“Yes but she can’t dance” said Jill
“Ahh,” said Ted, “but I didn’t say she had learnt to dance.”

  • Assessment should be aligned with our intended learning outcomes  (as in Biggs’ constructive alignment).
  • Students need to ‘get it’ and therefore they need to be prepared for assessment by ‘rehearsal’ using formative assessment. You can’t expect most 1st years to write a good academic essay without first having given them the chance to practice academic writing and to get feedback on it (and that can be peer feedback).
  • Feedback should look back and then feed forward. “You have passed because…” (explain what they did well) & then “This is what you can work on to improve…”
  • Students often do not understand the discourse of assessment and feedback. The language of Higher Education can be like a ‘secret garden’ with students “trying to get into our heads” in order to figure out what we want from them. Therefore we need to be clear and explicit about what we expect from our assessment. We need to express our tacit knowledge.

“Students don’t know what feedback is and when they are receiving it! They don’t understand the discourse of feedback.” This rather fun advert highlights how some of our students can feel when they get their marks and feedback, “I must have said something right” but by implication I’m not sure what it was that I did right!

Mark recommends a model for helping students understand assessment and succeed.

Modelling—–>Joint Construction—–>Independence

  1. We start with modelling, showing students how to do it. This is demonstrating  our unconscious competence in this area. For example showing them how to write academically so they can see it in action (this could be as part of a lecture or seminar or as an online resource)
  2. Joint Construction is the formative assessment part. Students practice doing it together with us and their peers.
  3. Independence is when students are ready for the high stakes assessment. They are not just thrown into it to ‘sink or swim’.

This is then set in a context of scaffolding (similar to Vygotsky’s Zone of Proximal Development), student metacognition and dialogues (staff-student, student-student.) “We have a duty to help students succeed…even if some don’t want to”.

Some Questions to ponder

  • The ‘hurry-up’ curriculum, “do we try to do too much too quickly in the first year?”
  • Is Computer Aided Assessment good enough yet to revolutionise the efficiency of assessment?
  • Is our assessment purposeful? Does it lead to and allow for success?

Congratulations to Mark on an inspiring, interesting and educational presentation.

NB: all quotes are taken from the presenters at this conference under their section of the blog post unless referenced otherwise.