Where We Are Now; Or, Why We Have Never Been in Phase II

Because I was positioned as the DIS participant who would defend Phase II, I'll start there, and then work my way around to discussing the convergence of assessment, reflection, and transfer (which, by the way, is exactly where my interests are situated). 

Let me tell you how I became acquainted with Phase II. It was the spring of 2012, and I was conducting research on my program's portfolio assessment process, particularly in regards to its clearly flawed approach to assessing the reflective component of student portfolios. I won't rehash the whole experience because Joe and Bruce where in Writing Assessment, where I wrote frequently about that work, and KY read a draft of my study. Suffice it to say that the way Ed White discusses portfolio scoring at the bottom of p 584 aligns almost precisely with the way my program approached the matter: standard rubric, calibration sessions, benchmark essays, etc. Now, as a soon-to-be rhetoric and composition student/researcher, the role that reflection played in this whole portfolio process just didn't sit right with me. None of the faculty seemed to agree on why we assigned reflection, what it was good for, what it should look like, how we should integrate it into our classrooms, and so on. I set about attempting to change this, and as I was making sense of the data I collected on students' reflective introductions or reflective essays--depending on who you asked--my faculty mentor suggested I take a look at Ed White's Phase II. 

In retrospect I can say that I played Elbow's believing game a little too hard. I understood the practical difficulties a program faces in conducting portfolio assessment--time, money, faculty engagement, student and faculty resistance, and so on. I have been that uncertain reader trapped in a room with other uncertain readers over the course of three days, and let me tell you, it's pretty frustrating. So, my practical experience combined with my interest in reflection led me to focus (naively) only on those aspects of White's article. Asking faculty to care about reflection? Yes! Actually taking time to read and consider student reflection? Yes! It sounded like a great approach, and I played a big part in revising our assessment documents and practices to focus more heavily (though not exclusively) on what I understood to be reflection. 

Now, revisiting White's essay, I might be playing the doubting game a little too hard (if that's possible to do in this case?). Let me just run through a laundry list of the problems I see: 1) White is assuming that there was such a thing as "Phase I"--that all people in all places using portfolios used them the same way. Though he does admit that portfolios have different uses and purposes depending on the context (583-584), he doesn't seem to adhere to that understanding in this piece. 2) "Students" are all the same; there is only one class of student, and s/he seems to be a lazy trickster who indulges too heavily in "personal experience" and wallows in the "difficulty of writing and revising" (591). And when this student encounters Phase II, s/he is overjoyed and apparently says things like, "I never realized how much I have learned, since I was just interested in passing courses" (591). Ed White has three decades of experience that prove Phase II to be "the only assessments...that students genuinely find interesting, useful, and worth doing" (591). Which brings me to 3) To whom does White look for information about potential problems with portfolio scoring? Ed White (582). Also, this "means for scoring portfolios that has developed over the last few years..." seems to have no origin point. Who developed these means? Where? What's the context? Was there a study? Where's the data? 4) And, perhaps most problematically, the meaning of reflection in this piece is really difficult to pin down. For instance, in this passage White writes, "One particular strength of portfolio assessment is its capacity to include reflection about the portfolio contents by the students submitting portfolios. Most such assessments require the portfolio to open with a "reflective letter" or "cover letter" in which the owner of the portfolio comments about the products or the processes shown in it" (582; italics mine). Putting aside the issue of referring to the student as the "owner" of the portfolio, I have trouble understanding how what White is calling reflection actually is reflection in the sense/s that we've been discussing in this DIS. Here, reflection is about; later, "students should be involved with reflection about and assessment of their own work" (583; italics mine). Seemingly, the student is not the agent; s/he is not reflecting, but is merely participating in some kind of disembodied process that we're calling "reflection." But I think White's movement in this sentence from reflection to assessment makes sense: he is ultimately just describing institutionally prescribed self-assessment. Is that the same thing as reflection? I think the other readings from the past weeks can help us answer that question in the negative.

Leaker and Ostman really seem to describe a similar sort of process in writing about their experiences with PLAs: students are to reflect on their experiences and to render what they've learned institutionally recognizable. The similarities stop here, though. Leaker and Ostman are suspicious of ways that the institution co-opts student experiences and learning, of the way that student encounters with institutional assessment can harm them,  and of how this assessment process fails to respond appropriately to diverse student populations. What Leaker and Ostman come to recognize is that they must help students develop a vocabulary and a repertoire of rhetorical strategies with which they can make sense of and articulate their experiences in ways that the institution will recognize. Student learning is there already, and it can be carried forward through reflective exercises within the PLA context. The danger lies in what Dorothy Smith calls "institutional capture," articulated here by Leaker and Ostman: "Crudely put, the approach to alterity in this model is to encourage students to reflectively reframe it so that it becomes visible as academic credit" (15). Now, Ed White seems to view the institutionally-captured experience and learning as what we're going for when we ask students to reflect--we (Ed White) want students to articulate what they've learned using the language and values of the institution, and if they don't, perhaps they should retake FYC or leave the institution altogether. 

So, I have to say that I agree with Joe when he writes, "I think reflection is necessarily an articulation: in order to explore the self, to learn about ourselves as writers, and to develop our writing, we need to be able to articulate those ideas in order to manipulate, remediate, and revise those concepts." Reflection does seem to be about articulation, for ourselves and for others, i.e. Yancey's reflection-in-presentation. Yancey, Robertson, and Taczak (YRT, from this point forward) write about reflection and transfer in this way when they review transfer literature and emphasize in their own research the importance of students' developing a vocabulary for the discipline. They also write about the three-part role of reflection in their TFT course: 

(1) students learn reflective theory by reading about it; (2) students complete successive reflective assignments, including one accompanying every major assignment in which students theorize about key terms, writing processes and practices, and their identity as a writer, and (3) students engage in other reflective activities connecting reading, key terms, and assignments. (58)

Vocabulary clearly plays a key role in each of these components as students (re)articulate their theories of writing; however, students' experiences, learning, and identities are not forced into an framework of institutional outcomes. Students are not merely involved with "reflection"--they are reflecting, which ultimately leads to synthesis when they produce their theories. Joe asks what forms reflection-as-articulation can take, and I think my response would be, many. Gallagher gives us the example of the interface, which he does not directly connect with reflection, though it seems to me to be an example, especially in the context of KY, Stephen, and Elizabeth's article on eportfolios. Leaker and Ostman, and YRT give us another example: verbal or written reflections spurred by reflective assignments. And yet another: visual mapping in the TFT course of how key terms relate to one another, to other terms, experiences, contexts, and so on. Despite Bruce's skepticism on the matter, I would side with YRT and say that the literature demonstrates that reflection has much to do with transfer, that it does promote transfer, and that, as Kara has also argued elsewhere, it's necessary for transfer to occur. And, it seems that reflection in many forms can only aid in this process, though I would agree with Bruce that more research in multiple sites would help to support this claim.

In sum, here's where I am: reflection is crucial for transfer and important for ethical assessment. I believe that Wardle and Roozen's ecological model of writing assessment can help us to bring these three areas together, and hopefully we'll hear more from them soon on how the implementation is going at UCF. In the meantime, at the classroom level, assessment within the TFT course clearly depends upon reflection-as-synthesis, with this synthesis having been enabled through semester-long (re)articulations of a theory of writing. In these syntheses and in ePortfolios, we can see what students have concurrently transferred to and from other contexts. What we can't see is what happens after the TFT classroom over the course of a student's academic and professional/post-uni career/life. I believe the longitudinal portfolios that Wardle and Roozen describe would provide one way to trace this kind of transfer. We also don't yet know much about what happens with TFT in multiple sites (though that research is coming!), and we don't yet seem to know much about the roles that various "forms" of reflection play in the learning process. How might visual mapping interface with synthesis? with reflective assignments? How does this change when different materials and technologies are introduced? How do students of different racial, cultural, and economic backgrounds experience the TFT course, and does it privilege a particular way of knowing that troubles certain populations (as Leaker and Ostman describe)? We've already started drawing the map, but we have a lot more territory to chart in the coming years.

Glue

I’m glad I finally read Ed White’s article on Phase 2—it gives me hope that in fifty years, I can write about Phrase 3 portfolios where we use telekinesis to grade a portfolio without even putting eyes on one actual portfolio. And God willing, if my career was successful enough, it will get published in CCC.

In response to this week’s blog prompt and White’s Phase 2, I want to focus on a question I’ve had about reflection: what is it’s scope? Dr. Yancey’s book says explicitly that reflection is both a process and a product, and throughout the semester, I’ve been interested in that idea of reflection as a product. At the conclusion of this DIS, I think reflection is necessarily an articulation: in order to explore the self, to learn about ourselves as writers, and to develop our writing, we need to be able to articulate those ideas in order to manipulate, remediate, and revise those concepts. But what form can this articulation take? What is the scope of this articulation?  To Ed White (and even to some of the others we’ve read this week such as Leaker & Ostman and Allan & Driscoll) seem to focus on assessing the written text. White, while applauding the wonders reflection has to learning writing, has a fundamental misunderstanding of where this reflection is happening: to White, the reflection is only happening in the single written text.

Now, Gallagher’s article, I believe, is the glue that has given me (at least) the connection between reflection and assessment that I was looking for.  I really like his idea of interface as a space for performing context of reception and production. His case study of Brenda’s (his student’s) e-portfolio demonstrated how the visual arrangement of content in the portfolio not only was a way to show a user how to navigate the space, but as a way for users to inhabit her articulation, to inhabit her reflection.  If we understand the e-portfolio, itself, as a reflection(-in-presentation), then Gallagher offers a pretty interesting definition of reflection (at least in terms of how it functions in assessment): the reflection is meant to perform a context for the viewer’s reception.  As Gallagher points out at the start of his piece, successful assessment is always aware of context—so the reflection offers and performs that context.  Reflection aids in assessment by providing that context (I realize this is kinda reductive, but I’m just thinking this through. I think the reflection probably has more functions for assessment than just to aid in providing context, but I’d have to think through that a bit more.)

So we have reflection in a written form, in visual arrangement or design, but we also have people like Tony Scott who shows that interviews and talking through writing demonstrate a different kind of set of values than in written text—maybe that’s also reflection if they’re talking about their attitudes toward writing. And Kevin Roozen in his document-based interviews—the object or documents as points to reflect. Or Doug Hesse, who talks about the essay as reflection, but in a way, does the sign on the bridge embody a collective reflection? Or does it just prompt it? What about the Vietnam War Memorial? Does that embody a reflection or just prompt it? I’m not sure, but that brings us back to my question about the scope of reflection. If reflection is necessarily an articulation (as I’m saying), what forms can it take? That sorta where I’m at right now.  

Phase 2 = Profits!

At least I tried.  I really tried.  I read Ed White's "The Scoring of Writing Portfolios: Phase 2" and did my absolute best not to become hyper-critical and attempted to understand his argument.  However, when it was all said and done, all I could think of was this South Park clip (warning: I couldn't find this scene with the language edited out so there is cussing at the end):

You see, throughout this episode, the gnomes fail to ever explain Phase 2 no matter how often they are asked.  Essentially, they just keep insisting that stealing underpants will lead to profits.

In a similar manner, I could not in any way understand (much like Stan) how in the world Ed White's concept would provide any benefits short of expediting scoring processes and producing flashy data.

When it comes to the connection between reflection and assessment, I've come to the conclusion that it is a matter of the difference between proving learning and understanding learning through reflective writing.  These two might  not be as separate as my convenient little binary suggests; however, I do believe they provide some insight into the benefits/drawbacks of reflective writing becoming tethered to assessment.

White's "Phase 2" really seems to be about proving student learning.  Reflection, in this case, is used as evidence that a student has met or not met certain outcomes.  As White notes, "With the focus of the assessment on the degree to which the student's reflective letter demonstrates awareness of and accomplishment of the goals, portfolio reading can proceed relatively quickly and with high levels of agreement" (590).  "Phase 2," thus, seems more intrigued by proving that the students have met certain goals and less interested in understanding the learning process.  This type of thinking seems to be in league with standardized testing in that it focuses primarily on quantifying learning.

In contrast, the Leaker and Ostman piece seemed more focused on understanding learning.  Granted, PLAs are a method of proving learning as to get college credit; however, Leaker and Ostman's use of reflective writing seems more inclined towards helping the assessors and the students to understand their previous learning.  Through understanding what these students have learned in non-academic contexts, it can then be determined whether they have prior experiences that would equate to the learning that would take place in a traditional college classroom.

Reflection, for me, is a rather valuable assessment tool.  It can enable tremendous insight into the learning process, both for students and teachers/administrators.  Thus, I think it has a great deal to offer us in the future, but we must remain vigilant that we are using reflective writing for productive purposes (i.e. understanding student learning to improve teaching practices, having students use reflective writing to gain greater insight into their thinking, etc.) instead of using it as a way to demonstrate whether learning has indeed taken place (especially in the rather empirical way that White outlines, which seems to not mesh well with the genre overall).

As far as transfer is concerned, I think this is a trickier question since, well, transfer seems to be one of the trickiest facets of pedagogy to research.  I think reflection has much to offer transfer; however, I think understanding just how reflection can foster and promote transfer is a slippery line of inquiry.  I'm in agreement with the authors of Writing Across Contexts who seem to desire to replicate their research in a variety of contexts. Replication--in my opinion--will be key to understanding how reflection might promote transfer.  We need to see how such classes, with a heavy emphasis on reflective writing and key terms, work in a variety of contexts.

We appear to have the question of "why use reflective writing?" pretty much figured out in my estimation, yet we still have a long way to go as to the "how?" question.  Replication and collaboration seem to be key here; since reflective writing taps into the concept of metacognition in such a profound way, research in this area will not be easy nor cut-and-dry.  Examining reflective writing across contexts seems to be key.

Reflective writing has a lot to offer us from a pedagogical standpoint and an assessment standpoint.  Yet, my greatest fear is the reduction of such a complex phenomenon.  Our approach to reflective writing needs to be nuanced, much like the practice of reflective writing itself.

Transfer, Assessment, and Reflection

Assessment and reflection, because of portfolios, have been linked for over 20 years, and transfer and reflection for over 15. Given our readings for our last meeting, where are we now in these two important areas? What do we know, and what do we need to know? And how does this connect to reflective writing and to pedagogy?

"The sight of the little madeleine had recalled nothing to my mind before I tasted it": Memory Work and Reflection

I want to pick up with some the points Joe raises, not only because my late post just so happens to follow his, but also because of some reflecting I've done on my own as of late. I'm just going to throw out a lot of not necessarily collected ideas and hopefully write myself into something productive.

Memory is something I've been interested in for quite some time, which probably explains why I spent two years of my life reading In Search of Lost Time. For the past year or so, I've wanted to revisit the novel, but have, understandably, found myself lacking the very thing the novel requires: time. Given a small surge in free time as of late, I've restarted the novel.

Additionally, my father is visiting for about a week, which means that we spend a lot of time reminiscing over what has been--not necessarily the "good old days," mind you, but the old days. And, as Joe points out in his post, these conversations are often spurred by objects: my father notices a picture on my wall of his holding me as an infant, which sparks recollections of coming in from second shift and staying up with me while my mother worked third shift. Or, not related to objects necessarily, having a drink on the screened porch during a rainstorm prompts my father to recall times he spent as a child at his grandmother's house. "It's amazing," he says, "that I remember it so well and I was only a child."

And this is the connection to Proust's narrator: objects do have tremendous effects on the narrator--an uneven patch of cobblestone, a Madeleine, a particular book in the library of an old friend--but these effects are sensory: the feeling of losing one's balance on the cobblestone, the taste of the Madeleine in a certain kind of tea, the feeling of seeing the images and reading the book in a particular place. One of the many themes of the book, at the most elementary level, is just that: we can recover lost time through object-interactions (transactions?). Rain felt, heard, and seen through a screen porch can transport an old man back his youth. So, memory, yes; sight-alone, no.

For me this is related to Roozen's interviews which, as Joe also points out, rely on the interviewee bringing in an object to spur recollection. Sometimes space and time between interviews helps with that. Roozen's main point seems to be that we can learn much from object-based reflective interviews that we might otherwise fail to see by, for instance, merely having participants draw their processes and surroundings (Prior and Shipka). Reading about this method is timely for me because I considered in the research proposal I just completed for research methods that Roozen's method (as outlined in earlier articles) could be useful for having students walk me through the maps that I ask them to make in class. But now I wonder if having the map is enough (or if having what seems like a "flat" map is enough). Couldn't other objects related to students' literacies be useful for mapping and a reflective interview that is aiming to learn how students compose theories of writing?

Obviously the kinds of recollections to which Proust's narrator speaks are not feasible in an interview setting. But I do wonder if a task that Yancey outlined in Reflection in the Writing Classroom might come close to inspiring this kind of reaction: Yancey writes about asking students to bring in one object associated with/representative of their literacy, and then to write 250 words about it. I wonder if this type of object-use could be helpful as students compose theories of composing. I am certain that looking to other modes could enrich this process. I have considered having students keep something like a commonplace book (print or digital) for collecting artifacts related to composing. Could this commonplace book help students to compose more robust theories and visual representations of those theories? Could this work enrich the kinds of reflective interviews that Roozen describes? Is it feasible to organize a class around this kind of work (commonplace book, concept mapping, reflective interviews/conferences with each student)?

Ultimately, I suppose I'm saying that memories are clearly an important part of the reflective process, so doesn't that warrant some kind of attention to the ways that memories are evoked, processed, and re-presented through reflection?

Reflecting Across Contexts: Objects, Public Memory, and Silence (or lack of it)

 I don't think I know nearly enough to make a judgement about whether theories in reflection are moving in the right direction or progressing, but I am left with a few questions after reading The Rhetoric of Reflection.

I want to start with a memory:

About a month and a half ago, I drove up to Maryland to get my annual emissions check for my car--as some of us know, that didn't work out as planned: I hit some tire debris in North Carolina, spent a night in Rocky Mount, rented a car the next day, and drove home to spend a week at home waiting for my car to get repaired. In that week with my parents, we had a lot of time to spare; so, we decided to visit Washington D.C., only a 30 minutes drive from my parents place in Annapolis. 

Let's pepper in some scholarship before I tell you why this is relevant.

From The Rhetoric of Reflection, two chapters in particular piqued my interest: Doug Hesse's and Kevin Roozen's. First, Roozen's piece offers, I believe, something about reflection that I haven't yet seen in what we've read so far: he discusses the idea (somewhat indirectly) of objects and memory. Roozen discusses (what he calls) two basic approaches to reflective interviewing--I'm focusing on the second: "reflective interviews that focus on the writer’s writing or action with a particular text or set of texts. These kinds of interviews involve using texts or artifacts to stimulate the writer’s recall" (emphasis mine). Later, he conducts these kinds of interviews with Terri and allows space between interviews for her "to locate and retrieve materials." She then brings those materials to the interview. Her and the interviewer use those objects as a point to bring attention to her tacit knowledge, the goal of the interviewing. But I began to think more broadly: I began to ask the question about the role that objects or artifacts play in reflection, in particular memory. Memory, though not mentioned explicitly, seems to be of central focus in reflection--Kara and Liane talk about the 360-degree reflection which requires students to draw attention to memories beyond the writing that students have done in their course, but I'm not sure memory is alluded to. But Roozen has shown that objects--Terri brings things like poems or technical charts--can further help draw out memory.

But what about other objects like scrapbooks that contain life-histories? Gifts from an ex-girlfriend that you've kept but you may not be able to articulate why you've kept it? Or a memorial at Washington D.C.?

Hesse's chapter, though talking about 'the essay,' is further talking about something much wider than the written text: I was particularly drawn to Hesse's example from the bridge: "Last Death from Jumping or Diving from Bridge, June 15, 1995." As Hesse explains, this is a sign posted in a bridge in Iowa--one that he has a special connection to because of his connection to the area that this bridge is located. He then discusses the importance of the essay written by Lia Purpura that talks about this sign: 

“'Jump' is ultimately about writing itself, the challenge not simply of figuring out what happened twelve years ago on the bridge but also figuring out her own reaction and, finally, shaping all that figuring into a text worth caring about. To do this, reflective moments simultaneously undercut and impel: 'I don’t know why' (89).  'All this time I’ve been thinking it over, trying to figure out how to read the thing'” (emphasis mine)

Of course, Hesse is talking about how the essay is demonstrating a kind of reflection, but what about the actual sign? Doesn't the sign, itself, invite reflection? The sign seems to operate as a memorial, a reminder of a particular event and the bridge's connection to that event. But the memory is not from an individual's experiences, but a public memory, one that the whole town can share through generations. 

So, now I return to my visit in Washington D.C.. As my parents and I walked from memorial to memorial, we noticed a trend: there were certain signs that attempted to prompt the attendees to take a moment to reflect. Here's a video of our visit to the Lincoln Memorial:

We obviously noticed the irony: there was a sign for quiet, but that was probably the loudest and most crowded place we visited, but the point is they want a moment of silence--why? They say for respect, but I think this is a prompt for reflection. You hear stories (myths) about when presidents would visit the memorial late at night to do some thinking (or if you're Nixon, visit it to talk to the people). In fact, I noticed that most of the memorials in DC were designed for reflection: for example, the reflecting pool that leads up to the Lincoln Memorial and reflects the image of the Washington Monument: 

But a more obvious example of reflection would be the Vietnam War Memorial. 

 One of the distinguishing features of the memorial is that it literally reflects your image over the names of those who were killed in the war. I don't have a video of our visit to this memorial, but I will say that walking through was a surreal experience: no one made a noise. The entire day, DC was crowded with people visiting for the cherry blossom festival, but here everyone recognized the need for silence. Now, as I reflect on the experience, I want to raise questions about the role that these memorials play in everyday reflection--or reflection that occurs without a teacher present. Instead, we have objects present that prompts you to draw back into your memory. But it's not so much your memory but a collective and public memory. What's happening in that kind of reflection? And why do we see that kind of reflection as important? But further, what role does silence play in reflection? To these questions, I don't know, but I think these are the questions I hope to see played out in the next round of reflection scholarship.