This article explores the advantages and trade-offs of two paths to the development of electronic portfolios - using generic tools and using customized systems. The analysis assumes that the goal of electronic portfolios is to stay focused on the quality of work by a learner and the valid alignment of their work to the standards and goals of education. A comparison of the "best of all worlds" scenarios for both approaches shows that the two approaches each have important strengths that need to be weighed during the process of program planning.

Portfolios have been around a long time. Educators used them long before the digital age, which suggests that perhaps the new media and medium of electronic portfolios, in part, can be looked at through lenses from the past. For example, in the past, portfolios were assembled from collections of work stored in boxes or three-ring binders and now are stored in digital form. Electronic portfolios, according to Barrett (2002), are essentially a new kind of container and can be developed along two paths. One path uses generic tools (GT), such as word processing, HTML editors, multimedia authoring tools, portable document format (PDF), and other commonly used productivity tool software. The second path uses "information technology" customized systems approaches (CS) that involve servers, programming, and databases.

What are the advantages and trade-offs of these two paths if the goal of electronic portfolios is to stay focused on the quality of work by a learner and its valid alignment to the standards and goals of education? By what criteria can the two approaches be compared?

The significance of these questions arises in more than the technological sense of which methods best preserve the heart of traditional creative, reflective portfolio work. Because the new medium is so flexible, it can easily be used for practices that can work counter to the traditional goals of portfolios. For example, once they are digital and more easily stored and searched, e-portfolios might be used as high stakes gatekeepers, like standardized tests of today. Just because digital portfolios can be used this way, should they? Our discussion of directions in e-portfolio development should help clarify the differences in the political, human, and technological dimensions of e-portfolio decisions, identifying the trade-offs, strengths, and weaknesses.

Criteria for Comparisons

To begin the conversation about pluses and minuses of each kind of path, Barrett (2002) offered the following. The ability to aggregate data for assessment is counted as a plus for CS and a minus for GT, and the low start-up and maintenance costs count as a plus of GT (leaving the question of whether it is a plus or minus for CS). She also pointed to Mary Diez' (1996) conception of the portfolio as "mirror, map, and sonnet." These metaphors hint about possible criteria for the comparison of GT and CS approaches. The mirror concerns the portfolio's reflective nature that allows learners to see their own growth over time. The map includes the portfolio's ability to aid learners in planning, setting goals, and navigating the artifacts learners create and collect. And the sonnet points to the portfolio's role as a framework for creative expression, encouraging diversity within a template or structure for thinking about work and presenting it to others.

Also suggestive of criteria for the comparison of GT and CS, Davis (2002) noted the benefits of e-portfolios as providing flexibility in three important respects:

1. Organizational flexibility.
2. Display flexibility of content and ideas
3. Ability to connect content to various schemas for representation in multiple ways, such as to standards, key concepts, interdisciplinary connections as well as linking works to one another.

To these we could add Communications and Collaboration Tools, since e-portfolios provide not only a new container for work, but also a new process for producing, improving, and sharing it. Beyond these issues are considerations of the ways that e-portfolios actually change thinking and action through both the media and the processes of supported reflection in learning (Jonassen, Peck, & Wilson, 1999). These considerations provide a starting point for discussing the pluses and minuses of GT and CS as directions in electronic portfolio development, a list of criteria for comparing approaches.

• Planning & Goal Setting
• Framework for Creativity
• Communications
• Collaboration Tools
• Reflective Processes
• Connection Capabilities
• Organizational Flexibility
• Display Flexibility and Transportability
• Data and Information
• Start-up Costs and Maintenance

Examples of several new e-portfolio approaches and projects were assembled at the first ISTE Assessment and Technology Forum held at NECC in 2002 and are available online as documentation of the forum's Gallery Walk (http://www.ideasconsulting.com/portfolioessentials/index.html ). In the following section, we mention some of the projects from the forum as illustrations of the criteria we describe.

Moving Toward a Rubric

To facilitate a comparison on each of the criteria, imagine a continuum with pure CS at one end and pure GT at the other, recognizing that this is an abstraction for the convenience of debate. Simplistic definitions of the two approaches used in this discussion are as follows:

• Learners construct their own portfolios using the Generic Tools (GT) approach using whatever digital storage space they have available.
• Using the Customized Systems (CS) approach an educational organization or a company provides an online database environment that provides a structure and server space for learners to store and organize their portfolios.

A pure CS, one might imagine, does not require the learner to know anything about HTML, nor to actively organize or link work. It can be thought of as essentially a single on- or offline application that to some extent allows and depends upon external applications for the production of work products. We use the term "Application" in rubrics presented in this article to describe the CS approach and distinguish it from an educational program using the GT approach.

The imagined CS application maximizes cross-portfolio comparisons and, in general, seems more "top-down," controlled by an educational program. Much of the process in a CS system - the flow of work and creation of linkages and documentation - takes place within the application, with most of the work products being made externally to the application using generic tools. Assessment management systems tend to fall on this end of the spectrum, with the goal of creating a digital record of achievement, especially a record that program requirements have been met. In an educational environment controlled by accountability concerns, CS systems help administrators prove that their programs are producing the desired results. The Personal Learning Portal described in the ISTE Gallery Walk is an example of a GS approach.

A pure GT approach, on the other hand, requires the learner to learn and use multimedia tools and HTML or an equivalent language or process for producing the digital products and a final display or collection. Learners in a pure GT approach start with a blank slate and must construct an entirely original representation and expression of their work, resulting in unique collections that are difficult to compare from learner to learner. The GT approach can be thought of as an assortment of external applications called upon to construct the electronic collection and documentation of work. In general, this approach seems more "bottom-up," controlled by a learner.

Much of the process in a GT approach is in private work and face-to-face communications (with asynchronous communications as a possibility) within an educational program. The process takes place without any overarching application and utilizes many separate applications to create or document work for digital collections and to form the collection itself. We use the term "Program" in the rubrics to distinguish the GT from the CS "Application" approach. Examples of products of the GT approach abound in faculty web pages on the Internet, especially in higher education systems where the institution's primary support amounts to access for transferring files to a server. The Barrington Public Schools portfolio described in the ISTE Gallery Walk is an example of a GT approach.

Between these idealized extremes, many electronic portfolio projects in education are hybrids, borrowing from both approaches as they create their own unique solutions. In what follows, a rubric framework presents the essential criteria for both the CS and GT approaches, offering a way to see the e-portfolio development process as a series of choices about capabilities. A full discussion of the criteria, with developmental stages outlined for minimal and partially developed systems is forthcoming (Gibson & Barrett, 2003).

The set of criteria might be applied to an existing e-portfolio product or program to help raise questions about the purposes, supports, and desired next steps in establishing development directions. We first present a suggested rubric for each criteria, then we present a summary rubric for all criteria. We are interested in receiving feedback on these initial ideas for criteria and score points to assist e-portfolio developers in planning and implementing their own systems. Of special value would be references to sites that can serve as "benchmark" samples that illustrate the criteria.

Planning and Goal Setting

Based on Mary Diez's (1994) metaphor of the portfolio as "map," one benefit of an e-portfolio is its use as a tool for planning self-directed learning or setting learning goals, either by the learner, or an advisor, or both (see Table 1). The OLLIE Project shows a structured approach to professional development planning and monitoring in a CS approach, and can be contrasted with the open-ended Personal Learning Portal, as well as GT orientations such as the Wake Forest Professional Development Portfolio.

Framework for Creativity

Based on Diez's metaphor of the portfolio as "sonnet," does the approach allow learners to express their own creativity within the structure of the e-portfolio (see Table 2)? See the Collaboratory Project for an example of a framework for creativity using a CS approach.

Communications

Does the approach use telecommunications to facilitate communication between learners and advisors/faculty (Table 3)? See the John Hopkins University Electronic Portfolio for an example of a CS approach to online communications, and the Mount Abraham High School site for an example of face-to-face communications using a GT approach.

Friedrichs and Gibson (2003) described a framework and process of collaboration focused on personalization. Does the e-portfolio approach provide tools that support collaboration between learners and advisors or faculty (Table 4)? The Collaboratory ePortfolio and Personal Learning Portal illustrate two CS approaches to collaboration.

Reflective Processes

Based on Diez's (1994) metaphor of the portfolio as "mirror," the e-portfolio should provide an opportunity for learner reflections on artifacts as well as how they support multiple purposes (goals or standards). Does the approach support reflection to address multiple purposes and audiences (Table 5)? The Johns Hopkins University Electronic Portfolio illustrates an institutional commitment to reflection and assessment of e-portfolios in a CS approach.

Connection Capabilities

In what follows, a "schema" is meant as a framework that represents standards or other organizational constructs that are useful to organize the presentation of portfolio products. An electronic portfolio should take advantage of the linking capabilities of digital technology, to make easy connections between various work products, the learner's reflections or rationale statements, and the various schemas. How does the approach provide the capacity for the learner to make linkages according to the various purposes or audiences (Table 6)? Linkages in a CS system are illustrated by the Personal Learning Portal.

Organizational Flexibility

The flexibility of the e-portfolio's organization will determine its usefulness to meet multiple purposes. Is the e-portfolio approach flexible enough to facilitate the multiple organization frameworks (Table 7)? The Collaboratory ePortfolio illustrates a CS approach to organizational flexibility.

Display Flexibility and Transportability

The flexibility in being able to display the e-portfolio in multiple formats will determine its usefulness to meet multiple purposes. Is the e-portfolio approach flexible enough to display the work in multiple formats (Table 8)? Is the data only accessible in a specific system, or can the e-portfolio be transported to other formats? Most example sites illustrate this point.

Data and Information

E-portfolios used as part of a program evaluation process provide the opportunity to aggregate data across program objectives and learners. Does the approach address learner privacy and intellectual property issues (Table 9)? Does the approach facilitate the development of a comprehensive picture of each individual student as well as the capacity to aggregate data across an institution or program? Do the organization's needs for assessment data interfere with the learner's need to present a rich and individual representation of their growth over time? See the PASS-PORT project and the Certification Program Portfolio for examples.

Start-Up Costs and Maintenance

The e-portfolio approach needs to be cost-effective to be useful and used. What are the start-up and maintenance costs of the approach, from the viewpoint of both the organization and the individual learner (Table 10)? Does the system provide support to learners and faculty/advisors, as well as IT staff? The Minnesota e"Folio" project plans to make its system available for free.

A summative rubric for comparing e-portfolio approaches is presented in Table 11.

Which Is Better?

We hope the brief picture presented here helps show that both approaches have strengths, depending on the degree to which a particular criteria has been addressed by both application developers and the educational program. Either approach can stand alone, but they may be weaker for doing so. A CS approach by itself soon loses touch with the individuality of inquiry and expression of learners. A GT approach by itself limits its contribution to a program's validity, as well as accountability.

Which approach is better also depends upon the purpose and audience for the information within and connected to a learner's portfolios. Note that the word "portfolios" is plural, pointing to the fact that for a body of work by a learner there can be several portfolios or sharing collections, each aimed at a difference audience for different purposes. For example, in a private working portfolio, the audience is the learner and the purpose is to improve work over time. Some of the strengths and advantages of the CS approach are irrelevant to this portfolio, for example, comparisons across portfolios of other learners. Not only does the comparison violate the privacy of a private working portfolio, but also the information that could be obtained from other portfolios is unlikely to be comparable. Imagine, for example, trying to compare a budding composer's private working folio with that of a future lawyer; not only the content and intentions would be radically different, the symbolic notations and language would be incomparable.

Even within a single discipline area, when two people have agreed to try to represent the same subject using a common style and starting point, great differences in each person's private intentions are practically guaranteed to lead to divergent results. The divergence of thought and expression in the GT approach is a good thing for the expansion of knowledge through the free play of the creative mind!

An example from the CS perspective is evident when considering the impacts of a program on its graduates. For example, are candidates being well prepared to teach and lead in the schools and communities where they will find work? As educators frame answers to this kind of question and ask to see products of student work that demonstrate that we are indeed meeting such a challenge, common frameworks for development and assessment of work seem to be an obvious criteria. In a public portfolio demonstrating the impact of the program, some of the strengths of the GT approach are not helpful; for example, if individual creative expression makes determining the impact difficult or creates an unfair advantage or disadvantage for making that determination. Imagine assessing a lesson plan and a dance for evidence of an ability to plan instruction to meet the needs of all students. Not impossible perhaps, but such divergence soon stretches public credibility.

Adopting only one of the approaches rather than working to integrate them can lead some institutions to "short-cut the thoughtful process of designing an e-portfolio strategy and supporting technology" (Cambridge, 2002). The crux of the issue is whether the organization - department, school, college - is interested in teaching thinking and reflection through a process, or if those core concepts become lost in the many other requirements for accountability. E-portfolio planning teams need to realize the balancing act involved and stay vigilant! In doing so, portfolio processes can be included and valued as valid and reliable additional measures of student learning.

As Cambridge (2002) noted "Preventing what we value about the concept of portfolios from getting lost of high stakes assessment is different from insulating it from that process. High stakes assessment is not going to go away, whether we like it or not." Electronic portfolios or portfolio-like assessment management systems can and should "provide a powerful way to inject authentic assessment into a process that otherwise pays virtually no attention to what students and teachers value and the actual work they do." Reliable quantitative data can be produced in electronic portfolios "in a manner which pays attention to how students actually learn and helps put decisions about pedagogies and curricula in the hand of the professionals who are most qualified to make them, teachers and faculty."

Conclusion

The realistic alternative is then an interpenetrating combination of both approaches with appropriate caution about which approach is driving the process in relation to purpose and audience. A program and its developers cannot confuse systems accountability issues with portfolio reviews and feedback to learners. Nor can they afford to confuse individual expression with achievement, or privacy with isolation of information. These issues can be reconciled within a complex system in which roles and access are appropriately determined for different audiences for different times and purposes.

The benefits of taking on the challenge of integrating both CS and GT into a comprehensive e-portfolio development program are numerous. Rich pictures of learning in incommensurably individual expressions are as useful as highly comparable measures of performance and achievement and both are needed for a deep understanding of learners and a program's impact on them. Individual performances linked to commonly experienced program opportunities help tell the organization's story of impact better than either the numbers or the pictures by themselves. Use of both centrally held and widely distributed capabilities can strengthen all users of data to create information. Appropriate use of generic as well as customized system tools builds technological, critical, and creative thinking skills.

The emphasis on which tools and how deep they are developed varies with the context as well as the challenge facing the learner and their supporters. The success of that emphasis depends as much on how a program and its developer partners are communicating and codeveloping as on the static qualities of the generic tools or customized system approaches.

References

Barrett, H. (2002, June). Introduction to the forum. Paper presented at ISTE's Forum on Assessment and Technology, San Antonio, TX.

Cambridge, W. (2002). ITFORUM public comment. Retrieved January 20, 2003, from http://listserv.uga.edu/cgi-bin/wa?A2=ind0212&L=itforum&F=&S=&P=4133

Davis, H. (2002, June). Features of electronic portfolios. Paper presented at ISTE's Forum on Assessment and Technology, San Antonio, TX.

Diez, M. (1996). The portfolio: Sonnet, mirror and map. In K. Burke, (Ed.), Professional portfolios: A collection of articles. Arlington Heights, IL: IRI/Skylight Training and Pub.

Friedrichs, A., & Gibson, D. (2003). Personalization and secondary school renewal. In J. DiMartino, J. Clarke, & D. Wolk (Eds.). Personal learning: Preparing high school students to create their future. Lanhame, MD: Scarecrow Press.

Gibson, D., & Barrett, H. (2003, March) Electronic portfolios: Decisions and

dilemmas. A workshop presented at the annual conference of the Society for Information Technology and Teacher Education. Albuquerque, NM.

Jonassen, D., Peck, K., & Wilson, B. (1999). Computers as mind tools for schools (2nd ed.). Upper Saddle River, NJ: Prentice-Hall.