Certification programs rely on portfolio-based assessment certifications when the traditional exam-based evaluation method will not measure the qualification in a reliable and valid manner. The strength of a portfolio assessment certification program is that it can measure the application of knowledge and skills in a professional setting. As a result, portfolio-based certification programs are favored by those who prefer performance-based certifications, especially at senior career levels.
And yet, portfolio assessment certification programs are notoriously difficult to manage online. Unlike examination-based programs that often share common characteristics, portfolio assessment certification programs manage highly specific artifacts and evaluation criteria.
Where exam-based certifications rely heavily on high-stakes assessment, portfolio assessment certification programs rely on detailed evaluation of work product that varies from program to program.
Nonetheless, the benefits of putting a portfolio assessment certification program online are compelling. Applicants like them because no one wants to deal with paper anymore. Staff like them because they want a single place to track where applicants are in their process, and reviewers like them because they can create a uniform, reliable evaluation process.
So How Do You Make Your Online Portfolio Assessment Certification Program Successful?
Technology will play a significant role in the success for your program, as you think about your technology objectives, consider the following:
Ensure applicants have a positive experience that guides them through the process and answers their questions. Make sure the system provides feedback about missing items and prevents applicants from submitting incomplete applications.
Give staff members insight into the volume of applications, especially with respect to reviewer assignments.
Provide a standardized evaluation rubric for reviewers to assess portfolios so that the evaluation process is consistent.
Analyze reviewer responses for timeliness and bias.
With these considerations in mind, the right certification management system will support even a complex portfolio assessment certification program.
The Applicant Experience
The applicant experience is the aspect of the online system that offers the most options. Plenty of online form builders exist that can collect complex information, including uploaded documents and extended text responses. When evaluating a platform, consider the following requirements:
Saving incomplete applications: Applicants must be able to save incomplete applications, as they are unlikely to assemble all necessary information to submit everything at once. They need to be able to easily see where they left off so they can resume once they find all of the necessary materials. Programs that have extended application periods, sometimes spanning years, will become a familiar place for applicants as they return to their portfolios to report progress.
Timeline Dashboard: Present a dashboard that shows applicants how far they are in their application process. Show what they have submitted in the form of a checklist and keep a count of the number of requirements they have completed relative to their total requirements.
Clear instructions: Provide on-screen instructions and prompts. Well. . .we know no one reads instructions, but the first complaint someone will have if there aren’t instructions is that they aren’t there. Make sure you have a place to describe your policies and expectations. Where possible, explain the rationale for the requirements. You don’t want to overwhelm your applicants, but you want to anticipate questions they might ask your staff.
Using a rules engine: Manage applications using a rules engine so the online system can make sure only complete applications can be submitted. For example, applicants will want to know that their application meets structural requirements when they submit and not worry that their application will come back showing they didn’t complete some section.
Easy payments: Make sure payment is seamless. You don’t want people to go through the trouble to prepare an application only to have them give up because it is too hard to pay. Make sure to calculate any variable payments in the system, so they only pay the right amount. Use a PCI Compliant payment gateway to make sure all cardholder information is secure.
Convenient notifications: Keep applicants apprised of application status through on-screen messaging and notifications. Some applicants will check in on the system periodically to see if there have been changes to their status. Others may wait to be notified by e-mail or SMS messaging. Ensure the system can send automated messages regarding changes to application statuses, such as confirmation of receipt, request for clarification, or notification of acceptance or denial.
A well designed online certification management system with these capabilities will dramatically improve the applicant experience by making the process transparent and easy to use. Your staff will be happy with fewer customer support requests and the number of applications that slip through the cracks of a paper-based portfolio assessment certification program.
Staff Member Insights
Without an online system, staff members of portfolio assessment certification programs bear an enormous burden to orchestrate applications and reviews. Online certification systems remove much of the manual work, leaving staff to focus on management tasks, like assigning reviewers and serving as intermediaries between reviewers and applicants. Your system needs to make these functions easy. Consider the following requirements:
- A mechanism to assign reviewers from a list of qualified members.
- Monitor the status of review queues to determine if any reviewers are falling behind in their assignments.
- A means to sanitize reviewer requests for clarification. Reviewers may raise important questions of applicants, but staff need to be able to monitor communications and potentially alter correspondence to achieve greater standardization.
Some systems may offer more advanced capabilities, such as automated reviewer assignments. Programs with large numbers of submissions and complex assignment algorithms will need a system designed for automated assignments. As you evaluate an online certification management system, make sure to be specific about your assignment protocols.
Reviewer Evaluations
In our experience, reviewers for portfolio assessment certification programs have tremendous demands on their time. Applications are often complex, with highly technical content and extensive materials. In addition to the complexity of the review process itself, reviewers are often volunteers and leaders in their field, which is an additional demand on their time. Therefore, the online experience needs to harness their expertise and pinpoint the exact areas they need to evaluate. Consider the following requirements for an online review experience for a portfolio assessment certification program:
- Targeted reviews – Knowing that reviewers do not want to think about “a system” and just want to “do their jobs” means providing a system that directs them to the precise areas they need to evaluate. The system needs to show them what is on their plate, what their tasks are, and how to enter their responses. The system needs to reduce the number of clicks and remove administrative clutter.
- Anonymous reviews – The need to eliminate reviewer bias starts with ensuring that reviewers are not influenced by distracting characteristics of the applicant. This means hiding all aspects of the applicant not subject to evaluation. For example, removing names and omitting demographic information prevents gender or ethnic bias.
- Unambiguous evaluation scoring – Forcing reviewers to select from prescribed options, such as “Accepted / Not Accepted” or “Exceeds / Meets / Does not Meet” requirements or a points-based evaluation scheme ensures clarity and standard responses. This is key for ensuring the reliability of the assessment process.
- Inter-rater collaboration – This allows reviewers to question each others’ responses to achieve a common frame of reference for evaluation. Posting questions and requesting internal clarification can lead to a more defensible outcome.
- Monitored dialogue with applicants – The unstructured nature of portfolio assessments means that some submissions will contain ambiguity. With the right protocol for dialogue with applicants, programs can request clarity without jeopardizing the integrity of the results. For example, if supporting documentation omits mention of a critical part of the evaluation, but the applicant can find additional documentation to substantiate a response, reviewers can give applicants a better chance to put their best foot forward.
- Logged interactions – Knowing that interactions between reviewers and applicants or among reviewers could open questions of reviewer integrity, a system that logs all responses will make sure any communications are subject to standard review.
The right online certification management system will meet each of these requirements according to your priorities and specifications. It will give reviewers the sense that their time is valuable and that their technical contribution is appreciated.
Response Analysis
The final component of an online portfolio assessment certification program is to provide evidence that the program actively manages the impartiality of its reviewers. Capturing reviewer evaluation plays an essential part in that process, but programs will need data to support the lack of bias. An online certification management system will need to provide the following capabilities:
- Inter-rater reliability analysis to determine if reviewers demonstrate observable departures from the mean.
- Demographic cross-references to determine how applicant characteristics unintentionally impact outcomes.
This kind of analysis can only be performed if the system both collects data and provides a mechanism to report on the data. Look for a system that includes robust reporting and analytics. If you conduct this analysis in external data mining tools already, consider whether the system you are looking at can expose data from the system to your external tool.
The Online Certification System for Portfolio Assessment Certification Programs
The list of requirements described above may seem daunting, especially if you intend to build your online process from scratch. A custom software solution will be expensive and, at best, result in “version 1.0.” Your association management system will probably represent many of these capabilities as “within reach,” but when you dig deeper, you’ll probably find that many of your requirements around workflow and evaluation rubrics will involve costly customizations. As with all customizations, they will be brittle, jeopardize future upgrades, and cost to maintain over time. Instead, consider certification management systems such as LearningBuilder that offer the flexibility to support many different kinds of certification programs.
In addition to the functionality described above, you’ll find that they offer all sorts of additional advantages:
- Integration with external systems like association management systems and learning management systems.
- Robust re-certification and continuing competence processes.
- Configuration engines that allow you to make changes to the system without programming.
Your IT department may tell you that an online portfolio assessment certification is out of reach. Your AMS vendor may tell you it will cost an arm and a leg, but the right certification management system will give you what you ask for and more. Go ahead—explore what your portfolio assessment certification program would look like online in the right system. It will be worth it!
Heuristic Solutions provides software solutions for certification boards that can give you the tools to manage a robust recertification program while gaining optional efficiencies. If you are considering models for continuing competence in your certification program, consider a certification management system with that feature set built-in. To learn more, connect with us on our contact page or schedule a meeting with one of our expert team members.
