The 1993-94 academic year was a transition year for the Perseus Project. The four-year development cycle funded by Annenberg CPB officially ended, the project headquarters moved from Harvard University to Tufts University, and FIPSE funding was secured to develop and evaluate instructional exemplars for using Perseus. Although institutional demarcations were clearly defined, the transition from building a system to creating and assessing applications of the system is ongoing. During 1993-94, most of the effort of the developers went into working on Perseus 2.0, an effort that continues into the fall of 1994. Likewise, developers and the growing Perseus community continue to use Perseus for instruction in a variety of courses and in a variety of ways. This report focuses on how Perseus was used at sites that agreed to be part of the FIPSE studies and includes some additional data from other sites that provided data to the evaluation.
A variety of data were collected and used to assess how Perseus affects teaching and learning. Three sites were visited (Holy Cross, Tufts, and Wesleyan) and instructors and students were interviewed. An interview with an instructor from another site (Bowdoin) was also conducted. Instructors at two sites (Holy Cross and Wesleyan) kept written journals, and students at Holy Cross also kept written journals. Students at four sites (Holy Cross, St. Louis Community College, Tufts, and Wesleyan) completed written questionnaires. Instructors from six sites (Austin College, Ball State, Bowdoin, Holy Cross, Tufts, and Wesleyan) provided syllabi and assignments and participated in a two-day meeting in the spring of 1994 where they reported results and discussed strategies for the coming year. The Perseus ListServ also provided a public forum for Perseus experiences.
A descriptive summary of each site provides a context for understanding how Perseus was used, the types of learning and teaching Perseus engendered, and the problems and challenges it created for instructors and students. Together with the four years of previous evaluation results, these summaries provide the basis for defining models and raising issues related to using hypermedia materials and digital libraries in teaching and learning.
This course was offered in the spring semester, 1994. The ten students in the course ranged from sophomore through senior years and all reported using some type of computer at least a few times per week. Eight of the students took the course as a Greek language course and two as a Greek culture course. Students were responsible for reading assignments (language students in Greek, culture students in translation) and providing verbal commentary on the readings. They accessed two standalone Perseus workstations in the classics department and another workstation was brought to class for discussion and demonstration by the instructor. Students were asked to keep a written journal of all their work and impressions of working with Perseus. After the first week of the semester, three more workstations were made available in the Language Resource Center and students began doing word searches related to "freedom." About half way through the course Perseus 2.0 was made available. On the questionnaires, students reported using Perseus between 5 and 50 times (mean=27.9 times) for a total of between 10 and 80 hours (mean=40.6 hrs.).
Information Overload and Information-seeking strategies.
The text tools in Perseus (Greek and English word search, morphological analyzer, dictionary) were developed to support sophisticated philological research. Using these tools requires systematic searches of texts and careful filtering of results to determine which word matches (hits) are relevant to the theme or concept of interest and which matches represent incidental lexical patterns. This filtering requires time and patience as well as contextual knowledge that can be used for making relevance judgments or finding relationships. In his journal, Prof. Martin described the frustration students felt as a result of information overload. One student noted in the interview: "One of the problems I'm having right now is, last night I was doing a search on the connotations that Zeus the god might have with freedom, and when I went to do the search in the author Euripides I had it search for Zeus and it told me that it found like 300-400 occurrences and did I want it to list all of them and it wouldn't do it so I think there was a bug or something; and when I did a search for freedom there were 250 or something." When asked in the interview what was the worst thing about using Perseus, several students noted getting too many hits during word searches. One student said: "I think sometimes that the amount of information that I get sometimes deters me from actually researching, kind of like X said, I get like 500 instances and I think well I want to know about that but maybe I don't want to know that much about it. I find it hard then to narrow it down and its difficult because, sometimes I think that in order to narrow this down I would have to see every one of these instances to see which ones are important--and I don't know how to narrow it down so I get frustrated." Although several students wrote about the many "hits" the words "free" or "freedom" provided, the time spent sifting through the retrieved passages was, on reflection, useful. One student wrote: "The amount of information initially overwhelmed our research group. It took many hours to glean any meaning from the text references Perseus provided. But those many hours would have been many weeks if it was not for Perseus. In the end, Perseus gave us the ability to come up with an intelligent view of what Herodotus thought about freedom."
Students prepared reports with handouts for their research on how specific authors treat the concept of freedom. Prof. Martin wrote about using these presentations as opportunities to discuss information-seeking strategies that different students used and how these strategies fit into the scholarly research process. Strategies he mentioned included: use truncated word stems to broaden lookups; look for antonyms when searches are unsuccessful (e.g., slavery for freedom); use resources outside Perseus; discuss with the instructor; and use dictionaries or encyclopedias. He wrote: "Students learned something about investigating, and we also began to identify some extremely important themes and words for investigation of freedom of speech, thanks in large part to the students' being more imaginative than before in conducting their searches and not giving up when they initially get no or few results from their inquiry." He also notes that one student who is taking the class as a culture course is learning how to "read" vases and developing typologies by making her own interpretations of scenes. Thus, although most of the information-seeking strategies were related to text, this student developed investigative strategies for critically and systematically examining vases.
Clearly, conducting word lookups causes students to encounter information overload directly. Scholars digest masses of related information over many years and when using computer tools they can use their extensive background knowledge to assist them in filtering through information. These students were encountering overload without the benefit of extensive domain knowledge. In some cases, Perseus led them to explore outside wheat was required and thus develop more domain knowledge. Some students noted how word lookups led them to texts they were not familiar with and they had to develop more extensive knowledge about an author or cultural context to understand a word or passages. Thus Perseus opened up new challenges as well as opportunities. One student noted: "Like what I was saying about that play that I found, now I'll go and read it because I need to know about it--so Perseus pointed me in that direction, and even though it can't offer me that (the play was not in Perseus) it will provoke an interest."As technological tools become more pervasive in K-12 environments, these types of overload will not be quite so surprising to students, but instructors who plan to use Perseus in this manner are advised to prepare students by discussing information-seeking strategies, including those described above.
Discovery learning and serendipity.
Perseus' primary materials allow students to explore and discover themes and relationships on their own. Identifying examples of such discoveries has been a goal of the Perseus evaluation in the past and several examples were identified at Holy Cross. In the third week, Prof. Martin reported a discovery made during a class discussion (differences in how Greeks and Persians thought about freedom). His great excitement was tempered by his commentary on the students' lack of appreciation for the discovery. He wrote: "I find this terribly exciting; I don't think the students grasp the significance of what is happening. But for me this is the payoff after years of hard work and frustration: I discovered something new and significant from work done in class by students using Perseus." The following entries discuss his mild disappointment with the depth of analysis and illustrate a key challenge to teaching for critical thinking: Providing a rich environment for analysis and synthesis when students do not have a sufficient knowledge base to recognize relationships and higher order effects.
Prof. Martin wrote in his journal about one student's discovery as a result of accidentally searching Frazer's notes on Apollodorus rather than the Apollodorus text. "In the end, he happened upon a very interesting detail about freedom of speech, namely, the silence of Telephus that Frazer connects with homicide pollution in his notes. This point will bear further investigation on its substance; the whole episode also demonstrates how learning by discovery can happen in unexpected ways."
One student investigated the role of women in Greek society by examining vases. She described her strategy which began with an English word search (e.g., "wife") and systematic viewing of each vase having the word in its description. She reported spending about 40 minutes per vase and using the textual descriptions to augment her visual examinations. In a journal entry midway through the course she wrote: "The reason this is so hard is because I do not know how to teach myself something I can't read or contextualize. By the end of the semester I want to be able to look at any vase and explain what the general scene being depicted is." In an entry a month later, she reported "Having spent the past month staring blankly at hundreds of Greek vases, knowing little to nothing about the connection from one to the next, I have finally felt a slight yet meaningful revelation. I started teaching myself how to "read" vases with the hope that the socioeconomic differences between female citizens in Athens and hetairia would magically befall upon me. Until quite by coincidence, I had lunch with a friend and we discussed why women tend to date men who happen to be taller than them. Not an hour later, I was sitting in front of a Perseus computer looking at what seemed to be yet another nameless vase. Then it struck me, in "Malibu 86 AE. 293" the tondo showed a young man and a hetaira who are the same height. After careful analysis, I realized that not only are they the same height, but their eyes are at the same point, their feet are arranged in the same manner, and their arms make equal but opposite gestures. Basically, they were in perfect symmetry....Keeping the symmetry of the tondo in mind, I began to inquire why Brygos (the potter) would depict a hetaira and a young Athenian man as "equal." As a result, I looked at a vase illustrating the wedding of Peleus and Thetis. Sure enough, Thetis was depicted as sitting, visibly "lower" than Peleus....Through this chaotic yet highly beneficial research session with Perseus, my research has moved into a different direction. Currently, I am hoping to develop the idea of symmetry as it applies to women in Athens; whether the symmetry hetairia have with men in regards to their intellect can in any way be matched (on vases) to the symmetry wives have with Athenian men considering their citizen status." In the interview, she reported her panic at using almost the entire semester examining vases without discovering a theme to write about for the final paper until this discovery, but also reported that the discovery was for her the best thing about the course. In the final paper, she was unable to prove her hypothesis with evidence from vases in Perseus, but the process of discovery and research that she experienced illustrates what is possible when undergraduates are provided with models for scholarly investigation during class and given rich sources of primary materials to use outside of class. This example illustrates the potential for Perseus when students and instructors are diligent enough to endure the frustrations of exploration and critical reflection on that exploration. This is something that cannot be taught, but as the Holy Cross experience demonstrates, can be encouraged and modeled.
Greek Language Learning: Time on task or No Pain No Gain?.
In previous sites (e.g., Ball State, St. Olaf) students who used Perseus in Greek language courses reported mixed feelings about using Perseus for translation. Although students appreciated the time-saving advantages of looking up words with Perseus, some expressed concerns that the ease of lookup influenced their translation skills less forcefully than manual lookup. This issue was discussed directly and indirectly during the Holy Cross interview with students.
Some students spoke of the advantages Perseus offered for going beyond simple definitions of words. One student said: "We were using it with Herodotus and in doing the translation if you need to know a word you just click and you have the definition and you know everything about that word right at your fingertips. Then he had us translate--Professor Martin had us move to Aristotle, and Aristotle wasn't on Perseus and I found when I was doing the translations that when I would come to a word, whereas if I were preparing for any other class I would just look it up, know the definition, and be able to translate it in class. Now that I'm used to using Perseus, when I look up a word, I want to know more about what that word means, I want to know how he (the author) uses it in other places, and its frustrating when you don't have the capabilities of Perseus for that." He followed up by saying: "It makes you do more work, I think. It makes more work possible."
Another student expressed the opposite view, saying: "I've used Perseus just strictly to do translation before and now I'm using it almost just strictly for research and I think its an excellent research tool but I'm not sure I like using it for the translation part because I think it was almost too easy, I'm not sure if I learned as much as I would have if I had to, on my own, search out, you know, everything about the word. I could just click and it was all there for me. I'm not sure if I learned as much that way." When asked to elaborate, she said: "(inaudible) different (inaudible) of the screen it says you know this is the perfect participle or whatever so you know what form of the verb it is and you simply have to look it up, but without it you have to look up what form it is and (inaudible) the definition and figure out, you know, what usage it is--instead it just tells you right there, and I think you don't spend as much time finding it out."
To investigate possible physiology-psychology interactions, students were asked whether literally holding a book and touching the words helped them translate. Most comments revolved around viewing projected text rather than text in a book and the advantages of writing notes directly in a book. One student said: "It's hard to get used to, I mean, for me we're doing a lot of work in class with the computer and the screen is there and its hard to get used to translating with text on the screen and not be able to take notes right there in the book when you're trying to look at the screen. I think that was the hardest thing to get used to. I ended up bringing my book and just tried looking at the book even though we had it on the screen--it was easier for me." Other students agreed, although one student reported doing translation from projections (not computer) in high school. Another student noted that limited text-window size on HyperCard cards interfered with his getting the big picture of the speech in Thucyides. Other students agreed, one noting: "So usually when I have to use a big section I'll go get it in a book so I can see it all at once."
Another student stressed that studying the text and definitions was what he found most important and that the medium of viewing or recording were not significant. For him, the medium of viewing and recording became simply a matter of efficiency. He said: "I always have to write things down when I translate...so for me it became an issue whether it was going to be quicker for me to look it up in the dictionary by hand or the machine was going to do it quicker and it all depends on the efficiency of it; that is the main thing for me because I'm going to have to write out the information, you know the Greek word and have to add a little blurb about the form or the definition, and I'm going to have to look over it anyway." When asked if cut and paste was helpful, he continued: "Definitely, it doesn't matter if I've written it out or if someone else has written it out, as long as I've gotten something to look at. It is just a matter of what is the most efficient." This point of view generated some argument, some students complaining about reading projected text in a darkened room as opposed from a personal book and one student suggesting that simpler cut and paste operations would avoid some of the current cumbersome cutting and pasting into a text file.
Clearly, there were differences of opinion about how Perseus contributed to Greek language learning. On the positive side, Perseus enabled students to do things more efficiently and go beyond the definitions of words to see how the author (or other authors) used the words in other contexts. On the negative side, the awkwardness of cutting and pasting, the limitations of screen displays (size, legibility, eyestrain), and the sense that word lookups were too easy and didn't require the problem solving and reflection that enhances memory made translating with Perseus problematic.
The nature of the course was also an issue since translation was part of the course but not anywhere near as prominent as it is in traditional author courses. Although this was made clear to students before registration and during the course, it presented pros and cons. On one hand, it admitted a broader range of students than a traditional language course, while on the other hand limiting the amount of language practice desired by language majors. When one student noted that since he had little Greek language background he could not be an effective student in an author level Greek course without Perseus, other students pointed out that the course was not an author course that improved their Greek language skills. One student said: "I don't know about anyone else but my skills with Greek definitely haven't improved and they may have like atrophied, because we've been away from Greek so long." Another followed with: "Its more like a topics course." The first added: "Its not a translation course." Students agreed that this was made clear from the start of the course. The first student clarified his position by stating that it wasn't that he learned much Greek language but "Previously I think research like this was closed off entirely to [anyone except] people who had a very strong Greek background and luckily I'm in class with people who do so I get all that kind of information that comes verbally from them; I think it would be a lot harder if I was just on my own."
These comments illustrate what some of the tradeoffs are as instructors use Perseus to enable new learning experiences and admit broader ranges of students in language courses. Instructors who plan to use Perseus must weigh these tradeoffs and as was done at Holy Cross, ensure that students know what costs and benefits will accrue. This is especially important during the early stages of innovation adoption and should be considered along with the need to develop and support physical infrastructure.
Transition from Perseus 1.0 to 2.0: Maintaining Physical Infrastructure.
One of the clearest results of previous Perseus evaluations (e.g., Marchionini & Crane, 1994) is the need to develop physical infrastructure for technological innovation. Physical infrastructure includes equipment, software, space for furniture and equipment, networking and projection facilities, and technical support for laboratories and faculty offices. The Holy Cross experience illustrated the classical challenges of getting infrastructure developed as new workstations became available during the semester, the instructor regularly moved a workstation and projector from his office to the classroom, and the inevitable bugs and breakdowns occurred on machines shared by a wide range of student users. A special challenge at Holy Cross was the delay in acquiring Perseus 2.0 which had some of the required texts planned for the course. For those texts, students had to work strictly from paper texts. After Spring break, an alternative assignment was made to take advantage of Perseus 1.0. Students were individually assigned an author and asked to investigate how the author treated "freedom." When the Perseus 2.0 prototype arrived after spring break, a number of problems occurred that made it difficult to use and a good amount of time and energy was expended getting the system to work and dealing with both 1.0 and 2.0 variations. Many students noted differences between the 1.0 and 2.0 versions, although they eventually made the transition. One student wrote "After having tremendously frustrating difficulties with 2.0 last time, I went back and continued my research in the orators. I found the second time through much easier because I was more used to working with the 2.0 system." System upgrades are common in computing environments and students and teachers must learn to deal with them. However, it seems reasonable to recommend that instructors who are not involved with system development and beta testing think carefully about changing systems once a course is underway.
Questionnaire Results: An Aggregate View of the Holy Cross Experience.
Students at Holy Cross had more experience using Perseus during the spring 1994 semester than any other site known to the evaluation. Their responses to the questionnaire are therefore less likely to be influenced by novelty effects. Table 1 presents descriptive statistics for items on the questionnaire (see Appendix A for a copy of the questionnaire and a key to variable names and scales.) As can be seen from the summary, these students found Perseus generally easy to learn to use (mean 2.1) and easy to use (mean 1.6), and rated the features of Perseus quite highly. These students reported the highest ratings for helpfulness in understanding, confidence in finding information, overall value, and satisfaction of all students who completed the questionnaire, and all said they would use Perseus again. All of these differences were statistically significant at the .05 level. These results are highly encouraging in light of the good understanding they demonstrated for the tradeoffs related to Perseus use and challenges of access and transition they faced over the course of the semester.
Table 1. Perseus Questionnaire Results for Holy Cross
AGE COMPEXP COMPFREQ APPEXP PUSE PFEAT HOURS LEARN USE
N OF CASES 10 10 10 10 10 10
10 10 10
MINIMUM 17 2 1 2 5 3
10 1 1
MAXIMUM 22 4 2 5 50 11
80 3 3
MEAN 20 2.90 1.60 4 27.90
6.30 40.60 2.10 1.60
VARIANCE 1.78 0.77 0.27 0.89 306.77
6.01 738.04 0.54 0.49
ST. DEV 1.33 0.88 0.52 0.94 17.51
2.45 27.17 0.74 0.70
DOC IMAGES LOSTFREQ RECOVER LIKLINFO CONFID ASSNDIFF UNDERSTA INFOUSE
N OF CASES 4 9 10 10 10
10 10 10 10
MINIMUM 2 1 1 1 1
1 1 1 1
MAXIMUM 5 5 3 4 3 3
4 3 3
MEAN 3.75 3.67 2.10 1.80 2.10
1.90 1.60 1.50 1.70
VARIANCE 2.25 1.25 0.99 0.84 0.77
0.54 0.93 0.50 0.68
ST. DEV 1.50 1.12 0.99 0.92 0.88
0.74 0.97 0.71 0.82
VALUE SATIS USEAGAIN
N OF CASES 10 10 10
MINIMUM 1 1 1
MAXIMUM 3 2 1
MEAN 1.70 1.50 1
VARIANCE 0.68 0.28 0
ST. DEV 0.82 0.53 0
Summary Learning Effects.
Perseus was clearly the most unique physical attribute of this course, for example in the interviews students noted that Perseus caused them to consider computer applications more broadly and recognize the importance of computers in all fields of study and work. The most significant intellectual difference of this course was the attention to modeling scholarly research and students' critical assessment of primary materials rather than regimes of knowledge and skill acquisition that is typical of undergraduate education. This change of emphasis was illustrated by the students and the instructor at the end of the course.
One student reported: "I think I research things more thoroughly than I would otherwise and I draw more comparisons; say if I'm researching something like Zeus, I can check every single time its in there...and also because the archaeology is right there with the primary texts it kind of encourages me to see--to maybe look at vases to see what would be on the vases or what would be on the coins and kind of tie everything together--the primary texts and the archaeology." Another student said: "I think Perseus encourages a lot of original thought because it presents the material very objectively so you can draw your own conclusions." Another student reported using Perseus for an archaeology course.
The final project for Greek language students was to prepare their source book entries which were to include an introduction to a text passage and translation related to freedom of speech with annotations and comparisons to other passages found in Perseus. Prof. Martin wrote in his journal: "The papers are decent quality according to my experience. The hardest idea to get the students to accept was that I did not want them to try to figure out what I wanted and that I wanted them to do their own thinking....My conclusion, which is supported by the students' answers to the Perseus questionnaire that Gary gave them and on the Holy Cross evaluation forms, is that Perseus gave them a chance to learn how to learn as researchers in texts. They still need lots more practice at it to be sure. In theory they could have done it with books (although not very efficiently and not so easily in groups). In reality, however, Perseus gave them the boost they needed to learn to learn by discovering and then making sense of what they discovered."
Tufts became Perseus headquarters in 1993-94 and a host of logistical challenges were met. Equipment and staff were moved from Harvard, Perseus 2.0 work continued, and a flood damaged some equipment and diverted attention. In the midst of this, Prof. Crane used Perseus in a Greek tragedy course and a combined section of first and second year Greek language. Perhaps the most positive term to describe Perseus use at Tufts is "minimal." The press of getting Perseus 2.0 ready for other sites (e.g., see Wesleyan below) and making the transition to a new location made actual use of Perseus in classes at Tufts a tertiary issue. Although the intention was for students to use Perseus in the Greek language classes to assist in translation, problems with the morphological analyzer did not allow students to do much more than run some vocabulary drills the instructor created outside of Perseus. The Greek Tragedy course sponsored seven lab sessions for students to use Perseus as a resource for a short paper, although a computer virus caused many problems in the early sessions. Adam Lewis observed students in these sessions and noted the following difficulties in his notes: "getting the idea of links, noticing the Navigator, learning the function of the GoBack button, remembering to select a word in Greek before they chose a morphological tool to apply to it." He noted: "People tended, I observed, to be less likely to ask me for help if they were working in pairs or trios, but they also seemed to do better, more, and more quickly while working in groups as well." Of the 60 students in this course, only 32 turned in questionnaires, which illustrate a definite lack of enthusiasm for Perseus.
Tufts students reported the lowest level of computer usage (least computer experience, second lowest frequency of use) and the least experience with computer applications of students from all four schools that submitted questionnaires. They used Perseus an average of 1.4 times for an average total of just 1.7 hours, and used the least number of Perseus features. They thus used Perseus less than all the other groups. The Tufts students reported lowest ratings for ease of learning and second lowest for ease of use (there was likely not enough experience to tell the difference) and their usage patterns are reflected in the performance items of the questionnaire. They reported the second lowest ratings for confidence in finding information in Perseus, degree to which Perseus contributed to their understanding, value to their learning, and satisfaction. As with the experience at Wesleyan discussed below, this site illustrates that to be successful, innovations must be used frequently and they must be used easily.
Table 2. Perseus Questionnaire Results for Tufts
TOTAL OBSERVATIONS: 32
AGE COMPEXP COMPFREQ APPEXP PUSE PFEAT
HOURS LEARN USE
N OF CASES 32 32 32 32 32
32 32 31 26
MINIMUM 18 0 0 0 1
0 0 1 1
MAXIMUM 23 4 4 7 9
7 4 5 4
MEAN 20.41 2.06 2.25 3.41 1.38
3.28 1.70 2.74 2.27
VARIANCE 1.41 0.90 0.97 2.83 2.05
4.08 0.71 1.20 1.32
ST. DEV 1.19 0.95 0.98 1.68
1.43 2.02 0.84 1.09 1.15
DOC IMAGES LOSTFREQ RECOVER LIKLINFO CONFID ASSNDIFF UNDERSTA INFOUSE
N OF CASES 9 24 32 28 29 30 30 30 29
MINIMUM 0 2 1 1 1 1 1 1 1
MAXIMUM 4 5 4 4 5 5 5 5 5
MEAN 2.67 3.96 2.84 2.79 2.66 2.93 2.03 3.03 2.59
VARIANCE 1.50 0.74 0.72 1.06 1.52 1.51 1.07 0.79 0.75
ST. DEV 1.22 0.86 0.85 1.03 1.23 1.23 1.03 0.89 0.87
VALUE SATIS USEAGAIN
N OF CASES 29 27 28
MINIMUM 1 1 1
MAXIMUM 4 4 2
MEAN 2.72 2.33 1.39
VARIANCE 0.64 0.62 0.25
ST. DEV. 0.80 0.78 0.50
Prof. Antonaccio designed an archaeology course to take advantages of Perseus 2.0. Assignments were designed to lead students to use museum pieces, books, and Perseus to explore themes in heroic myth and to learn stylistic techniques used in different artifacts. Due to delays in delivering Perseus 2.0 and then problems getting it to run on the campus Novell network, most of the Perseus experience at Wesleyan was related to developing infrastructure. As was the case in sites in previous years, physical infrastructure dominated the first year experience. Much of the electronic mail logged over this semester related to solving problems with Perseus on the network. The instructor, her teaching assistant, and the campus technical coordinator worked scores of hours loading and reloading the thousands of Perseus files, upgrading the server, and inventing ways to deliver Perseus over the campus network. One fruitful product of this semester was a set of procedures for using Perseus on a Novell network. This experience will be helpful to other users who use such a network on their campuses.
The "window of opportunity" metaphor is often applied to innovations to encourage a broader audience of early adopters (get in on the ground floor). All "windows" will eventually close but if the innovation causes frustration or high costs, the window will slam shut and those adopters will be hesitant to adopt similar innovations in the future. This caution is illustrated by comments recorded in electronic mail from one student to Prof. Antonaccio. In a message dated February 25, a student described her first experience with Perseus with exclamations and optimism. "I spent a good deal of time viewing the different photographs of sites in Greece (the Acropolis, etc.) and felt as though I was being given a personal tour right on the screen! On occasion, however, when I tried to call up a particular photograph, Perseus would stubbornly insist that "there is not enough memory room currently available--close some windows first" and no matter what I did, Perseus refused to show me the picture (rather temperamental!). Overall, it was a great introduction to this amazing program, temperament and all!" Although the system was not fully functioning for this newcomer to Perseus and computing, the novelty and expectation tempered the frustration. Almost two months later, in a message dated April 16, the same student wrote: "Perseus is being a bad boy acting up again (either that or I'm doing something wrong). I am trying to get into the vase collections to find examples of art depicting Perseus slaying the gorgonian medusa since I am doing my paper on Perseus, but the program keeps telling me that it is "unable to read" my command and then it asks me whether I want to "debug" or "cancel" and no matter which I choose it refuses to cooperate with my requests. I have tried everything from coaxing it to threatening it, to no avail....Grrrr!!!!! Any suggestions?" Clearly, patience wears thin when an innovation continues to frustrate users. Another student wrote in a message with the subject "this is crazy!" the following: "Perseus is being temperamental. Some texts he refuses to show us. Some pictures don't exist and I will not sleep for the next three days. Well, maybe a little. My point is that I'm doing my best." Although students had positive things to say about the course in their email notes, notes were received from eight students in the last week of classes reporting Perseus problems, asking for time extensions due to Perseus problems, and asking for technical advice.
The frustration of students is evident in the questionnaire data summarized below.
Table 3. Perseus Questionnaire Results for Wesleyan
TOTAL OBSERVATIONS: 16
AGE COMPEXP COMPFREQ APPEXP PUSE PFEAT HOURS LEARN USE
N OF CASES 16 16 16 16 16 16 16 14 14
MINIMUM 17 0 1 2 0 1 1 1 1
MAXIMUM 20 5 4 8 8 9 20 4 4
MEAN 18.88 2.63 1.75 4.19 3.06 4.19 4.84 2.64 2.64
VARIANCE 0.78 1.85 0.87 4.16 4.60 5.10 24.86 1.32 1.02
ST. DEV 0.89 1.36 0.93 2.04 2.14 2.26 4.99 1.15 1.01
DOC IMAGES LOSTFREQ RECOVER LIKLINFO CONFID ASSNDIFF UNDERSTA INFOUSE
N OF CASES 5 15 15 15 15 15 15 15 15
MINIMUM 2 3 1 1 1 1 1 1 1
MAXIMUM 5 5 5 5 4 5 3 5 5
MEAN 3.60 3.60 2.87 2.60 2.67 3.07 2.07 3.47 3
VARIANCE 1.30 0.69 1.84 1.83 1.38 2.07 0.78 2.41 1.14
ST. DEV 1.14 0.83 1.36 1.35 1.18 1.44 0.88 1.55 1.07
VALUE SATIS USEAGAIN
N OF CASES 15 15 13
MINIMUM 1 1 1
MAXIMUM 5 5 2
MEAN 3.33 3.07 1.46
VARIANCE 1.38 2.21 0.27
ST. DEV 1.18 1.49 0.52
Wesleyan students reported the second highest level of computer usage and the highest experience with computer applications of students from all four schools that submitted questionnaires. They used Perseus an average of three times for an average total of just under five hours, the second smallest amount reported even though Perseus was expected to be a major part of the course. Not surprisingly, the Wesleyan students found Perseus most difficult to use, were the least confident that they could find information in Perseus, reported the lowest levels of Perseus contributions to their understanding, gave the lowest ratings of its value to their learning, and were least satisfied with it. In short, these students were eager to learn the content and capable computer users but were fully frustrated over the course of the semester by hardware and software problems fully beyond their control and the control of the instructor and teaching assistant. Although this instructor and these students may eventually look back on their pioneering experience with pride, the costs of testing new technology must be recognized by those considering participation and by those responsible for assessing their performance. For junior faculty, administrative support is essential and for students, patient and tolerant instructors are a must.
Ball State represents the best example of Perseus adoption in an entire department. Building on the experiences of Prof. Moskalew and Prof. Payne in the past three years, other faculty have been using Perseus in their courses. It is important to note that physical and intellectual infrastructure to support Perseus has been developed over a three year period. Many of the professors have Perseus-capable machines in their homes or offices; there are two Perseus stations in the classics office work area; two campus labs have 4 and 8 Perseus workstations respectively with Perseus-aware attendants; and all classrooms are wired to allow the Perseus laser disc to be used in demonstrations.
In the 1993-94 year, Prof. Shea and Prof. McGrath designed and team-taught an upper level classics course Classical Myth and Theory in which Perseus was used for demonstration and by students in their projects. Although Ball State was not visited during this year, Profs. Moskalew and Shea participated in the spring retreat and reported on their use of Perseus. In her written report (Perseus and Me: The CC 305 Experiment), Prof. Shea described the origins and outcomes of the course (CC 305 is title Classical Myth and Theory). Based on rising enrollments, two sections (30 students each) of this upper level classics course were offered in the Spring of 1993. The opportunity of two sections and the fact that many students had some previous experience with Perseus in the introductory 105 course allowed Profs. Shea and McGrath to design team-taught sections that emphasized "collaborative learning", "increased, aggressive participation in the learning process", and creative thinking." Perseus was used for demonstrations related to mythic iconography and students were required to work in groups to develop Perseus paths to illustrate a thesis. These paths were presented by the group to the entire class using an LCD projector and videotaped.
In describing the outcomes, Prof. Shea noted: "Every group made a path and manipulated the computer with accuracy and even ease, but, in the end, we has asked to too much. The combination of an unfamiliar approach to a hallowed subject (they want to be told mythology and little else)...tension in group dynamics (has everyone done his/her share?)...the pressures of public speaking PRESERVED for strangers to see...the complexities of manipulating complicated and expensive apparatus that's not yours--and all for a grade in a required course--help!" In describing the drop in satisfaction scores for this semester, she noted: "Moreover, satisfaction scores will probably continue to suffer until we graduate all those who had learned to expect just a major research paper and exams/quizzes in 305: our bunch were only reluctantly in the avant-garde of the multi-media explosion."
The institutionalization of Perseus at Ball State is due to several factors. First, there are multiple faculty who use Perseus (Profs. Moskalew and Payne have been using it for several years in the CC 105 courses, Profs. Shea and McGrath are now using it in an upper level course, and other instructors have invited Perseus demonstrations in their courses). Second, these efforts have strong administrative support (Prof. McGrath is the group chairperson and the University President has given strong support to the use of technology in teaching). Third, the physical infrastructure has developed over time to support Perseus in core courses and in advanced courses with a large number of classics majors and minors.
Students at Bowdoin experienced a highly innovative use of Perseus because Professor Smith was developing the Perseus atlas throughout the academic year. He offered a course: "Geographic information systems in archaeology" that introduced students to geographic information system (GIS) tools and led them to apply these tools to archaeological problem solving. Twelve students ranging from freshman to seniors and with a wide range of computer experience were enrolled in the course. The importance of such tools to archaeology was captured in Professor Smith's comment in the interview: "...inherently, by their nature, all archaeological material and data are spatially oriented, geographically, and so when we have new technologies for spatial information, the question comes up does that affect the way we think about and approach this body of stuff that we work with."
Assignments and some exams were accessed through the Mosaic World Wide Web client and made use of the GRASS GIS system. Students worked in teams of two or three and spent a substantial part of the course learning to use the network and GIS tools and how to analyze data using graphical (e.g., histograms) and statistical (e.g., correlations) methods. In the first half of the class students completed weekly lab assignments outside of class that averaged 2-4 hours to complete while focusing on archaeological issues in class sessions. Students applied these tools to the geography of the Mediterranean region and explored archaeological problems suggested by Prof. Smith. The following essay assignment illustrates the integration of GIS skills and an archaeological problem.
"In the classical Greek world, funerary monuments are normally located outside of city boundaries; they often line roadways, just outside the city. One theory argues that visibility from the approach to the city is more important than location of the monument in relation to either the city or the roadway. Describe as fully as possible how you could approach this theory with a GIS. What information would you need? What data types would you use to represent the information? What kinds of analysis would you carry out? What GIS operations would you use to perform your analysis? (You may either describe this in terms of a generic GIS, or specifically in relation to GRASS.)"
Two learning events illustrate discoveries made as the class worked on the two objectives of learning to use GIS tools and investigating archaeological problems. First, a dramatic learning event related to the technology itself was described by Professor Smith. In learning how the GIS data is visualized and understood, students represented the same data set in different ways (e.g., examine different spectral level scales for a site, using different filters to enhance characteristics such as edges). In working with data for Macedonia, the network of canals and highways and the highly contrasting agricultural fields are practically invisible with the raw data display but appear in vivid detail when edge-detection filtering is used. Later, students were able to using multiple spectral bands to do accurate land-use classification that distinguished the subtle difference between forests and agricultural areas.
Second, in a subsequent lab activity the class was looking at city states that strike coins. Professor Smith had noted that there seemed to be a correlation between states that struck coins and altitude in one region. Student groups investigated other regions and the results were pooled. All results demonstrated statistically significant correlations between coinage and altitude--city states at high altitudes did not strike coins and vice versa. This discovery would have been impossible or highly unlikely without the GIS tools and Professor Smith is developing an archaeological theory to explain the results. Thus, students were able to actively participate in the original research of a scholar/teacher as part of their undergraduate study. When asked if he believed whether students appreciated the significance of this experience, Smith said: "I think so--yeah. In their projects, some of them are taking aspects of some of the things we have looked at in some of the lead-up labs; one group is essentially doing a direct continuation of what we found in that lab and in about two weeks I will be going to a New England Ancient History Colloquium in response to a call for papers on coinage in Greek city states and I will try to deal with some of these questions and that all came up in the course of the semester and so I told the students in class that this was essentially a reaction to the material that you put together; they got all excited about that."
He went on to describe how students are publishing on the Internet. "One of the things that they really responded to, and I think has really been effective in making them feel like they are doing good stuff is this scrapbook of snapshots that I've put on the webserver at Bates. And particularly since I can just look at the access logs and tell them, gee on this Sunday afternoon there were people from seven different countries that looked at the snapshots from the lab that you finished up Thursday!
Some wanted to know how to give their friends at different campuses instructions on how to look at this."
Enormous amount of personal effort on part of the instructor with very little technical support. This was particularly challenging since complex software and different platforms (Unix and Macintosh) were used integral to the course. This experience demonstrates, however, what is possible with persistence, patience, and collaboration as new technologies are applied to traditional problems and issues in classics.
Prof. Fuller of Washington University St Louis, taught a classics course at St. Louis Community College in which Perseus was used.
Table 4. Perseus Questionnaire Results for St. Louis CC
TOTAL OBSERVATIONS: 43
AGE COMPEXP COMPFREQ APPEXP PUSE PFEAT HOURS LEARN USE
N OF CASES 43 43 43 43 43 43 43 43 43
MINIMUM 16 0 1 0 1 0.30 0 1 1
MAXIMUM 59 5 4 7 15 13 90 3 3
MEAN 23.12 2.37 2.49 3.65 5.30 5.96 7.81 1.84 1.60
VARIANCE 83.39 0.95 0.83 3.14 8.93 6.96 190.89 0.47 0.53
ST. DEV 9.13 0.98 0.91 1.77 2.99 2.64 13.82 0.69 0.73
DOC IMAGES LOSTFREQ RECOVER LIKLINFO CONFID ASSNDIFF UNDERSTA INFOUSE
N OF CASES 38 43 39 38 39 39 39 39 39
MINIMUM 1 1 1 1 1 1 1 1 1
MAXIMUM 4 5 5 4 5 5 5 5 5
MEAN 1.87 4.16 2.41 1.89 2.46 2.23 2.41 2.51 2.82
VARIANCE 1.20 1.66 0.88 0.69 1.41 1.18 1.09 1.20 1.84
ST. DEV 1.09 1.29 0.94 0.83 1.19 1.09 1.04 1.10 1.35
VALUE SATIS USEAGAIN
N OF CASES 39 39 36
MINIMUM 1 1 1
MAXIMUM 5 5 2
MEAN 2.33 1.82 1.17
VARIANCE 1.02 0.84 0.14
ST. DEV 1.01 0.91 0.38
Although not part of the formal evaluation, Prof. Bonefas used Perseus in her "Topics in Greek Civilization" course and became an active participant in the Perseus Listserv. As a result of her reports and commentary, she was invited to the spring retreat and presented examples of her students' work. A series of eight assignments led students to follow Perseus paths, make notes using the Notebook feature, and design their own paths. Students were encouraged and sometimes required to augment their Perseus paths with secondary materials outside of Perseus (e.g., Assignment 5). In some assignments, students worked in teams and were assigned topics. Prof. Bonefas also created seven assignments that took advantage of Perseus in her Greek History and Civilization course. These assignments illustrate a sequence of assignments that introduces students to the features of Perseus and leads them to use Perseus as an integral resource in learning about Greek culture (assignments are available in the Perseus ftp archive).
In her demonstration, Prof. Bonefas showed how students used multiple software tools (e.g., graphics packages, presentation packages) to integrate Perseus resources into their presentations. The experience at Austin College illustrates what is possible with creativity and hard work; and the advantages of integrating Perseus as one of the "ordinary" resources for a course rather than some "special" resource.
As Perseus continues to influence individual students, teachers, and courses, it also has begun to influence the institutions, curricula, and communities that involve these units. The results from 1993-94 can be grouped into three general categories: teaching, learning, and institutionalization and community.
Demonstrations continue to be effective and low cost applications of Perseus. Student and faculty comments, questionnaire data, and ListServ discussion all demonstrate that seeing images and maps together with the texts provides a richer and more memorable learning experience for students. The costs are minimal since only one workstation and a projector is required, only the instructor needs to learn how to use Perseus, and careful planning can maximize class time. The importance of using Perseus as a resource for customized learning modules was taken a step further this year by the extensive (and ongoing) discussion on the ListServ about how to make electronic slide shows that integrate Perseus images with images from local collections. Although copyright issues must eventually be clarified, the current fair use rule may provide coverage for instructors who "spontaneously" create presentations that take images from Perseus and other digitized image collections.
Instructors make substantial investments of time and effort in acquiring Perseus and learning how to use it; acquiring and managing laboratories; designing demonstrations, lectures, and assignments; grading electronic assignments; and learning how to effectively integrate it into classroom interactions. These investments were highlighted again in this year's evaluation and must be considered by adopters. As the evidence above illustrates, in addition to the time and effort, there may be lots of frustration and student evaluations may suffer. This is particularly important to junior faculty in institutions where teaching is highly prized. Incorporating the products of efforts to use Perseus (e.g., assignments, logs) into teaching portfolios is one action that may be useful in this regard.
Existing examples of using Perseus to model philological research as part of the teaching process were extended this year at Holy Cross and at Bowdoin. Professor Martin used Perseus to purposefully model his own investigative strategies as he and his students examined how the ancient Greeks used words related to freedom. Moreover, he enabled and encouraged non-Greek language students to invent research strategies for critically "reading" vases. Professor Smith carried his own research efforts to apply GIS technology to archaeology to the classroom and enabled students to participate in an exciting discovery and participate in new forms of electronic publishing. These examples should inspire other instructors to leverage Perseus to take the intellectual risks of sharing the joy and frustration of their own research with their students.
The mechanical advantages offered by Perseus were reinforced in this year's data. Saving time in going to libraries and locating information (especially images) was commonly cited as an advantage. Defining Greek words was said to be faster than printed lexicons and string search capabilities were highly prized by some students. This year, there were three promising directions in the evaluation team's search for ways that Perseus enables new kinds of learning.
First, the Holy Cross experience of modeling research and critical thinking was quite successful. Students appreciated the research process and the difficulties it presents in terms of creating one's own problems and making one's own decisions. Although the philological assignments had been given in several sites in the past, this course made the process explicit by tying it first to a theme (freedom) within the context of Greek culture and secondly to the practice of philological and historical research. Perhaps the most essential outcome was that these students were able to discuss the research process and how challenging it is by contrasting it with what they usually learn in undergraduate courses--i.e., that they were able to articulate what they had "missed" as a result of creating their own "assignments" was itself an illustration of reflective thinking. Similarly, the Bowdoin students not only learned to use two powerful new technologies (GIS and WWW) but also shared in the excitement of making a new discovery in archaeology. It is likely that these students will long remember that course and the systematic search for relationships in data sets.
Second, more students were involved in using Perseus both as an information resource AND as a tool for presenting themes. A rich variety of content and presentations was observed. The experience of publishing work on the WWW and knowing that people around the world were accessing it was a powerful motivator for the Bowdoin students. The students at Austin College used Perseus and PowerPoint in creating effective presentations, and the students at Ball State University were able to present their Perseus paths within the boundaries of a videotaped production. In both these cases, many faculty would be impressed with the technical sophistication. These examples begin to hint at what will happen when all students have access to resources and tools before they come to college. The technical power must amplify the teaching and learning experience by making content more wide-ranging (good search tools for accessing large volumes of primary information) and presentations more vivid (integrated text and images as well as on-demand links).
Third, there were examples of technical literacy side effects. Students gained generic skills with computers and electronic information that should be useful to them in their future careers. At Holy Cross students learned about the Internet as well as Perseus workstations; at Bowdoin, students used a sophisticated geographical system to examine archaeological themes in ways that otherwise would have been impossible and the World Wide Web to access and present assignments; at Austin College students solved problems of integrating images, texts, and drawings in different electronic formats to produce electronic presentations; and at Wesleyan students experienced how computing is being applied in the humanities.
One final class of results is surmised from the Perseus ListServ and various Perseus-related presentations at professional meetings. During the September 1993 through May 1994 period, 427 electronic mail messages were logged by the evaluator. These messages total approximately 950K of text and include 295 messages to the Perseus List from scores of institutions. In addition to the active electronic community, Perseus continues to be featured at professional society meetings of classicists and educators (e.g., CAMWS, APA, ED-MEDIA) and in publications from both research and practice perspectives (e.g., ACM Transactions on Information Systems, Chronicle of Higher Education), and is listed as an "Important Computer-Based Project in the Humanities and Arts" in the "Humanities and Arts on the Information Highway" final report to the Getty Art History Information Program, American Council of Learned Societies, and Coalition for Networked Information.
Perseus has become an institution in its own right and is beginning to change the way research and education in the classics is conducted. The evaluation effort will document how Perseus continues to change post-secondary education as Perseus 2.0 becomes available and the Perseus community continues to develop new ways to use it in teaching.