Improving the quality assurance (QA) processes and acquiring accreditation are top priorities for academic programs. The learning outcomes (LOs) assessment and continuous quality improvement represent core components of the quality assurance system (QAS). Current assessment methods suffer deficiencies related to accuracy and reliability, and they lack well-organized processes for continuous improvement planning. Moreover, the absence of automation, and integration in QA processes forms a major obstacle towards developing efficient quality system. There is a pressing need to adopt security protocols that provide required security services to safeguard the valuable information processed by QAS as well. This research proposes an effective methodology for LOs assessment and continuous improvement processes. The proposed approach ensures more accurate and reliable LOs assessment results and provides systematic way for utilizing those results in the continuous quality improvement. This systematic and well-specified QA processes were then utilized to model and implement automated and secure QAS that efficiently performs quality-related processes. The proposed system adopts two security protocols that provide confidentiality, integrity, and authentication for quality data and reports. The security protocols avoid the source repudiation, which is important in the quality reporting system. This is achieved through implementing powerful cryptographic algorithms. The QAS enables efficient data collection and processing required for analysis and interpretation. It also prepares for the development of datasets that can be used in future artificial intelligence (AI) researches to support decision making and improve the quality of academic programs. The proposed approach is implemented in a successful real case study for a computer science program. The current study serves scientific programs struggling to achieve academic accreditation, and gives rise to fully automating and integrating the QA processes and adopting modern AI and security technologies to develop effective QAS.
The recent decades witnessed increasing involvement of quality assurance systems (QAS) in different fields of businesses. The reason for this is due to the great benefits that can be accomplished through implementing effective quality assurance (QA) processes. Those benefits include improved performance and outcomes, effective usage of resources, increased revenues, … etc [
In academic programs, the core functions of QA processes focus on assessing the students performance towards achieving the predefined learning outcomes (LOs) and providing effective improvement plans at both course and program levels. The LOs are defined as the knowledge and skills the students will earn upon the completion of certain academic program. Thus, achieving the intended LOs is a very important indicator that can be used to evaluate students performance, and hence the academic program quality. In addition, the assessment results of LOs represents the most affecting factor be considered when establishing improvement plans for both programs and institutions [
The absence of well-defined and effective approach for LOs assessment and continuous improvement processes represents a major obstacle towards automating those processes and developing intelligent and fully-automated and secure QAS in the field of education. The lack of automation affects the data collection, normalization, and visualization processes. This prevents the utilization of the state of the art technologies in the fields of information security (IS) and artificial intelligence (AI) to improve the efficiency of the QAS and hence the quality of the academic environment.
Specialized organizations such as the accreditation board for engineering & technology (ABET) provides guidelines about QA processes to be followed by academic institutions in the fields of computer science and engineering to satisfy quality requirements and achieve the accreditation certificates [
This article has two main contributions that seek to close above-mentioned gaps and develop efficient QAS for academic programs. First, it introduces an effective and systematic LOs assessment methodology that ensures the accuracy and representativeness of the assessment results and shows, in great details, how to utilize those results in the continuous quality improvement at the different levels of the academic program. The proposed methodology is implemented in a real case study to verify its effectiveness. The research’s activities are performed within the context of ABET and NCAAA accreditation processes for the Bachelor program in Computer Science (CS) at Imam Mohammad Ibn Saud Islamic University (IMSIU). Such case study aims to assist the program’s QA staff in developing hands on skills related to the accreditation procedures through a successful experience. It is worth mentioning here that the academic program studied in this research has received full academic accreditation from both ABET and NCAAA.
The second contribution of this research is the development of well-designed, automated, and secure QAS that enables to perform relevant quality processes in efficient and reliable manner. Several state-of-the-art cryptographic algorithms have been utilized to develop effective security protocols that provide required security services to protect valuable information processed by the quality system.
Another important value of this research is that it represents a starting point towards the intelligent automation and integration of all QA processes in the field of education and allows to utilize modern technologies to improve the decision making and the quality of academic programs. Remember that utilizing AI and IS technologies requires efficient data (in digital format) collection and processing in advance.
The CS program, in which the study was applied, has official courses and program specifications, which include LOs at course and program levels respectively. All courses, including their course learning outcomes (CLOs), in the program curriculum are mapped to some PLOs. The achievement level of particular PLO can be measured through the average achievement levels of all CLOs aligned with that PLO in certain courses used in the assessment process. Each course is assigned to a course coordinator who is responsible for managing the course delivery process and coordinating the relevant tasks between instructors of the same course. Every instructor prepares a course report (CR) illustrating CLOs achievement and the course evaluation survey (CES) results for his own section(s), while the coordinator prepares the comprehensive course report (CCR) showing overall CLOs assessment results at the course level, improvement plans, and the survey analysis results. The quality assurance and development committee (QADC) manages almost all QA and accreditation tasks in the CS department.
Section 2 of this article presents the literature review, the proposed LOs assessment methodology and QAS are introduced in Sections 3 and 4 respectively, Section 5 presents the results and discussion, and finally the conclusion is presented in Section 6.
Researchers have conducted many studies to recognize and standardize the QA procedures and accreditation processes for academic programs in different fields of study [
Researchers in [
The proposed methodology is depicted in
The outputs of the preprocessing stage include the PLOs-CLOs mapping scheme, in which all LOs at the program level are mapped with associated CLOs within courses that are used for assessment purposes. Courses are mapped to the PLOs at Introductory, Practiced, or Advanced levels. The PLOs-CLOs mapping scheme is the cornerstone for evaluating the students’ performance. The proposed approach ensures more accurate assessment of PLOs, since it relies on real students grades as taken from quizzes, labs, exams, projects, and other direct assessment methods. Moreover, it eliminates the need for using specific KPIs and rubrics for each PLO, and thus reduces the complexity of the assessment processes. The majority of previous assessment approaches use estimation results and only selected courses at advanced level. This may compromise the accuracy and not provide realist view about the students’ performance.
In the second stage, a table of specification (ToS) is developed for each course. The ToS represents useful tool for evaluating the achievement level of LOs at both course and program levels. The ToS is depicted in
The QADC develops an adjustable version of ToS and sends it to coordinators. At the beginning of each academic semester, the course coordinator, in collaboration with course instructors, develops customized ToS file for the course and shares it with all instructors teaching the same course, so each instructor can use it separately for his section. Thus, the ToS is customized for each course and filled separately for each section under certain course. As can be seen from
In the third stage, the improvement plans are developed at the course level. This stage is important to ensure that the continuous quality improvement is maintained for every course taught by the program. The process of developing the improvement plans takes inputs from previous stage, represented by the levels of attainment for CLOs, and the indirect assessment results obtained by CES. The results of the section-wise CES and ToS assist the instructor to determine strengths, and weaknesses for his own section(s), and hence contribute in the development of effective improvement plans considering every course instructor’s experience. The suggested improvement plan by each instructor is written in the CR, and then submitted to the coordinator to prepare the final CCR. The coordinator aggregates direct and indirect assessment results for all sections and compile them to prepare the final results and report them in the CCR. Then, the coordinator develops a mutually agreed on course improvement plan.
In the third stage, the improvement plans are developed at the course level. This stage is important to ensure that the continuous quality improvement is maintained for every course taught by the program. The process of developing the improvement plans takes inputs from previous stage, represented by the levels of attainment for CLOs, and the indirect assessment results obtained by CES. The results of the section-wise CES and ToS assist the instructor to determine strengths, and weaknesses for his own section(s), and hence contribute in the development of effective improvement plans considering every course instructor’s experience. The suggested improvement plan by each instructor is written in the CR, and then submitted to the coordinator to prepare the final CCR. The coordinator aggregates direct and indirect assessment results for all sections and compile them to prepare the final results and report them in the CCR. Then, the coordinator develops a mutually agreed on course improvement plan. The preparation of the course improvement plan takes three parameters, which are the overall CLOs achievement results, the CES results, and the course instructors’ opinion based on their experience related to the course delivery activities.
It should be highlighted that ToS uses actual students’ scores obtained by the different assessment methods in order to ensure more accurate and realistic results for the students’ performance towards achieving the CLOs. Another strength point for using the proposed methodology is that ToS and assessment methods are unified for each course and applied to all sections and students. No sampling technique is considered in this approach. This contributes in improving the accuracy and reliability of the assessment results. The ToS represents an initial step towards fully automating the assessment and QA processes, which has been done through the proposed system. Once the course improvement plan is approved by the concerned committees in the department, necessary actions are taken at the course and program levels to implement the plan in the next academic semester. In the proposed automatic QAS, secure protocols are adopted to provide the security services for the QA processes included in this methodology.
In the fourth stage, the proposed CLOs-based improvement plans are reviewed and then approved by concerned committees in the department. In particular, the proposed plans are first reviewed by the curriculum approval committee to make sure that proposed changes to the course are scientifically valid and applicable within the program context. In the second review process, the QADC double check the improvement plans to make sure they satisfy the quality standards for ABET and NCAAA accreditations. Then, plans are submitted to the department council for final approval. Once the plan of a particular course is approved by the department, it will be incorporated to the program curriculum and changes, if necessary, will be made on the specifications. Again, the developed system in this article allows automatic and secure implementation for those QA processes.
The fifth stage in proposed methodology pursues the LOs assessment processes at the program level. It uses the results of CLOs assessment, using direct methods, as inputs to the PLOs assessment process, in which the overall average achievement level is calculated for each PLO. The assessment processes in the fifth stage of the methodology specify the achievement levels for male and female students separately and the combined achievement level for each PLO. This allows to analyze and understand the students’ performance accurately and realize any possible variation between students’ achievement of PLOs based on their gender. This technique is appropriate for academic institutions, which have two separate environments for male and female students.
The proposed methodology allows to discover any weakness related to the students’ performance towards achieving the PLO, and trace that weakness to find its root. For example, the students’ performance in PLO# S1 is relatively low (59%), which represents a weakness point. The students’ performance can be analyzed to find out the courses and their CLOs that led to this weakness. As a result, the courses CS221, CS310, CS340, and CS445 represent the sources for the weak students’ performance in PLO#S1 as depicted in
The fifth stage receives inputs from secondary sources, which are the graduating students and faculty members’ surveys. This is an indirect assessment of PLOs attainment, which gives deeper understanding of students’ performance from the different perspectives of stakeholders. Unlike other assessment approaches, the proposed methodology uses the graduating students survey instead of alumni surveys.
The outputs of the fifth stage represent inputs to the sixth stage, in which comprehensive improvement plans are developed at the program level in collaboration with relevant committees, such as QADC and curriculum development A considerable part of the planning effort focuses on uplifting students’ performance towards achieving the PLOs.
The seventh stage is the final stage, in which the proposed plans are reviewed by the department council. If approved they will be documented in the APR and incorporated to the program and courses specifications to feed the first stage of the next round of implementing the proposed methodology. The next section presents the design and implementation of the proposed secure and automatic QAS.
In this section, we present the design and implementation of the proposed QAS. The security protocols used to provide CIA and non-repudiation services are illustrated as well. Although some steps of the system design and implementation may sound straightforward, they are essential to enable effective data collection and processing and hence achieve desired improvements. They also open the door for further utilization of technical advances in the field of AI and IS for the sake of improving the quality of educational programs.
A conventional QAS should grant privileges and allow each type of users (e.g., instructors, coordinators, Quality members, program development team, and manager) to perform certain tasks according to their roles. Each Instructor views, enters, retrieves, and updates data related to his own section(s), while the coordinator can be granted access to all sections under one particular course, so he can extract the CCR. Quality members and program managers, on the other hand, are granted wider privileges in order to perform the quality tasks assigned to them. An important aspect of the proposed system is that it enables collecting almost all data related to the program and students, such as evaluation results, surveys, statistics, corrective actions, CLOs and PLOs results, … etc. This allows to perform the assessment processes of LOs and other QA tasks automatically and accurately as discussed before. Moreover, collecting such huge and significant amount of data provides great opportunity for future research and analysis using advanced AI and data mining techniques.
In addition to above mentioned functionalities, the system maintains an outstanding level of security through extensive use of cryptographic algorithms. It provides the main security services, which are the confidentiality, Integrity, authentication, and non-repudiation.
To satisfy the huge and continuously changing requirements for the project of developing the QAS, we decided to use the scrum methodology. It follows effective iterative strategy during the development process, that allows to continuously test the system’s functionality, fix any deficiencies, and accommodate emerging requirements. After collecting the main requirements, the system development process considered first the most important priorities represented by the LOs assessment, and quality reports. Then, the use case diagram was developed as shown in
Prior to the implementation stage, the ER and DB diagrams were developed taking into account the major quality requirements. The main entities of proposed system and its Data Base (DB) design are shown in
Data processed by QAS and related procedures are very import and valuable. Stakeholders should be confident with regards to QA data and reports. Therefore, high levels of security should be considered when developing a QAS. The proposed QAS uses cryptographic algorithms and well-defined protocols to provide and maintain the CIA security services, and avoid the source repudiation problem. This improves the reliability of the quality reports handled by the system and ensures the transparency of QA processes and procedures. Two main security protocols were implemented; the secret keys sharing protocol and the data processing and reporting protocol, which are presented in
Ensuring the confidentiality and authentication is the main goal of the secret-key sharing protocol. It enables the program manager (PM) or department head to generate a private 256-bit key for AES encryption for each user and share the keys with intended parties in secure and efficient manner.
The secret-key sharing protocol can be described in the following steps: Each user should generate (using a true random number generator (TRNG)) his own RSA public (Pub) and private (Prv) keys with 1024-bits length at least. The main users are course instructors, coordinators (CO), Quality Assurance (QA) officials, program development (PD) officials, and the program manager (PM) or department head. The public key might be declared The PM generates the private 256-bits AES key, and signs it with his own RSA private (Prv_PM) key to generate the signature S_PM. Note that the signature guarantees that the key is really issued by PM and not altered in unauthorized way. This step can be repeated for each user if the manager needs to provide different keys for every group of users. For example, instructors in different courses might be given different keys so each instructor can encrypt and decrypt data and reports relevant to his/her course(s). The manager encrypts the AES private key and the signature (S_PM) using each user’s public RSA key. This encryption is repeated for each user in the system. For example, the key and signature of the QA official are encrypted using the Pub_QA. Encrypted keys can be then shared with relevant parties. Each party decrypt the encrypted message (AES Key || S_PM) to retrieve the key. Eventually, each user verifies the signature using the manager’s public key (Pub_PM). If the verification’s result is true, this means that key is reliable and issued by the PM, otherwise, the user should contact the PM to re-send the key and the signature.
In summary, the above protocol ensures that no one can access the data and quality reports else that authorized users who received the reliable key, which is ensured to be issued by the department head or PM. In the next section, the data and quality reports security (DQRS) protocol is presented and discussed.
The QDRS protocol is aimed at providing the CIA and non-repudiation security services. It enables different parties (instructors, coordinator, QA members, PD members, and PM) to process data, exchange and store the quality reports and related documents (including CSs and CRs) in secure and reliable way. In other words, the protocol securely performs the continuous quality improvement processes presented in this study. The QDRS protocol can be described in the following steps: Each Instructor inputs and retrieves data related to his/her own section(s) in the quality database. Data includes students grades, survey results, course statistics. Corrective actions, analysis, discussions, plans, and other parts of the CR are entered by the instructor for his section(s) as well. The Coordinator then processes data for all sections under his course and extracts the CCR. The processing includes, for example, calculation of overall CLOs achievement levels, the course survey results and other required processes for the CCR. The CCR should be mutually agreed on by all instructors in one particular course. The coordinator uses the highly secured SHA-512 to get a course digest or fingerprint that represents the CCR, updated CSs, and other course documents. The coordinator signs the digest using his own RSA private key (Prv-CO), which was issued in the previous protocol. The result is the signature S_CO. The coordinator encrypts the course documents and the signature S_CO using the AES private key, which was securely communicated using the key sharing protocol. Then, he sends encrypted documents [course documents || S_CO] to the QA official. The QA official decrypts the documents, using the private shared (AES), and verifies the signature to make sure that the reports and documents are sent by the coordinator and not altered by unauthorized party. This also prevents the source (CO) from denying his responsibility about those documents. After reviewing the documents and adding necessary reports, the QA official performs the steps 3–5 (using the Prv-QA in step 4). Then, encrypted documents are sent to the PD official. The PD official performs steps 6–7, and review the documents. Before sending the documents to the manager he performs steps 3–5 (using the Prv-PD in step 4). Again the manager performs the steps 6–7, review the documents, and then performs steps 3–5 (using the Prv-PM in step 4). Eventually, documents are stored in securely on the college servers. Note that after approval (digitally signed by the PM) from the manager, the course documents; including updated specifications if any, are considered approved so relevant parties can start implementing the improvement actions stated in the CRs.
The next section briefly describes the software implementation environment for the proposed QAS.
The proposed QAS was implemented using Java programming language. We used MySQL to develop the database. Graphical user interfaces (GUI) were also used to enable the interaction between the user and the system. Several cryptographic libraries were used to implement the aforementioned security protocols. The next section presents and discusses the results obtained
This section discusses the results of implementing the proposed QA approach. It introduces the gained benefits and the continuous quality improvement process. Assessment results and improvement plans can be categorized into course and program levels. The following sections illustrate both levels.
A sample of CLOs assessment for one course, which is data structures (CS242), was presented in
Examples for improvement plans that were developed as a response to CLOs assessment results are the development of unified class and lab tutorials of the course and re-organizing the course assessment methods to include more lab assignments and exams. This contributes significantly in improving the students’ performance especially in CLOs under skills domain.
The majority of current assessment methodologies depends on developing KPIs and rubrics for each PLO and then assessing students’ performance by giving him a number from 1 to 5 for each KPI. They also consider only sample of students in selected sections. Such approach does not give accurate and comprehensive view on the real students’ performance towards achieving the LOs.
In our proposed assessment approach, the PLOs achievement levels are calculated after obtaining CLOs achievement levels for all courses and their corresponding sections.
It can be observed from the table that students performance towards achieving PLOs# S1 and S2 is below the minimum acceptable level, which is 65%. This indicates that uplifting students’ skills relevant to those LOs should be considered as a priority when developing the program improvement plans. The table shows a comparison with the previous year’s results and the new target for next year.
In order to develop effective improvement plans the course reports, including CLOs and indirect student assessment results should be reviewed and taken into account, especially in courses that have low achievement of S1 and S2. Some examples of the improvement plans, that were developed based on the analysis of PLOs assessment results, include adopting research seminars, improving the program’s curriculum, introducing more lab oriented courses, and re-organizing the contact hours.
Learning domain | PLOs | PLO’s achievement for previous year (%) | PLO’s achievement for previous year (%) | Target for next year (%) |
---|---|---|---|---|
K1 | 66 | 68 | ||
K2 | 71 | 77 | ||
K3 | 80 | 80 | ||
S1 | 63 | 58 | ||
S2 | 53 | 53 | ||
S3 | 62 | 66 | ||
V1 | 70 | 73 | ||
V2 | 68 | 70 | ||
V3 | 71 | 71 |
The indirect assessment methods are used to assess students’ achievement of LOs and other relevant issues at both course and program levels. At the course level and for each section the CES indirect assessment is used to assist instructors to develop effective improvement plans for the course. The graduating students survey represents the second indirect assessment in the proposed approach. Unlike the practice used in previous studies, which relay on the alumni survey, this methodology uses a survey that targets students at the level of graduation. It aims to develop more accurate and realistic view on the students’ performance. This is because students at the graduation level are not affected by other outside factors yet, including the learning and experience earned from the work environment and many other sources. Also, students at this level are mature enough to provide realistic perception of the extent to which the PLOs are attained. Therefore, the use of graduating students survey as an indirect assessment for students’ outcomes is more appropriate than alumni survey.
The proposed approach includes systematic processes for continuous quality improvement at both courses and program levels. The flowchart of the continuous improvement process is presented in
Academic institutions seek to get accreditation and improve the quality of their academic programs. This requires adopting an effective QAS that ensures the accuracy and reliability of LOs assessment processes, and enables the development of efficient quality improvement plans. To address deficiencies in current QASs, a systematic LOs assessment and continuous quality improvement approach was developed. The proposed approach guarantees the accuracy of the assessment results and provides well-defined processes and procedures to utilize them in the continuous improvement processes. Such systematic approach is indispensable as a first step for fully automating and integrating the QA processes. In addition, the current study represents a guide for academic programs that aspire to obtain accreditation.
This research also introduced an automated and secure QAS that reduces efforts and performs relevant QA tasks efficiently and safely. The proposed system uses two security protocols and utilizes state of the art cryptographic algorithms to provide confidentiality, integrity, and authentication services. The source repudiation problem is also eliminated through the use of RSA digital signature algorithm. Furthermore, the system enables to efficiently collect and process valuable data that can be arranged later in datasets and used for advanced researches investigating the utilization of AI technologies to support decision making and improve the quality of academic programs.
Author thanks the Computer Science department in the College of Computer and Information Sciences at Imam Mohammad Ibn Saud Islamic University for providing the data required for conducting this research study.