DOI:10.32604/cmc.2021.014445
| Computers, Materials & Continua DOI:10.32604/cmc.2021.014445 | |
| Article |
Analytic Hierarchy Process for Evaluating Flipped Classroom Learning
1Department of Business Administration, College of Administration Sciences and Financial, Imam Ja’afar Al-Sadiq University, Baghdad, Iraq
2Department of Mathematics, College of Education for Pure Science, Al-Muthana University, Al Muthanna, Iraq
*Corresponding Author: Husam Jasim Mohammed. Email: Dr.hussam@sadiq.edu.iq
Received: 21 September 2020; Accepted: 29 October 2020
Abstract: In reality, the flipped classroom has gained popularity as a modern way of structuring teaching, where lectures move from in-class procedures to digitally-based assignments, freeing up the debate, and practice exercises class time. Therefore, it is essential to implement and analyze a way of teaching that will improve student performance. The paper aims to develop a model of the method of teaching science in Iraqi schools, and to assess whether teaching flipped classroom affects the achievement, motivation, and creative thinking of students by using the methodology of Multi-Criteria Decision Making (MCDM) in the Analytic Hierarchy Process (AHP). The AHP approach includes several steps, including setting assessment criteria and their weights, and by assessing the methodology of the flipped classroom as compared to the conventional cognitive learning process. An experiment was carried out in Iraqi secondary schools to examine the attitude of the students towards the subject of Chemistry. The findings have indicated that the students and teachers favored flipped classroom learning more than conventional cognitive learning. The study took the following parameters compared to the traditional approach: teaching techniques, learning flexibility, teaching aids effectiveness, student participation and working environment. This paper indicates that the teachers in Iraqi schools will be able to improve and do more preparation to shift towards flipped learning in the classroom.
Keywords: Analytic hierarchy process; flipped classroom; criteria weights; comparative analysis
Since its beginning, the learning systems at schools and universities have adopted the traditional education system. The students are educated based on teacher-centered learning, entirely through lecturing, laboratory work with the ‘chalk and talk’ and tutorial sessions in a traditional classroom [1,2]. Reference [3] stated that, apart from the mid-term test and quizzes, the majority of the lecturer’s time in the conventional class was spent on lectures as well as solving problems of a textbook sort, with the solutions copied from the board by the students. Since 2000, the academics have developed modules for the subjects delivered to the students in order to change the way of learning and teaching. The aim is to shift from a lecturer-centered to a student-centered learning, allowing a clear method in covering the course schedule as well as involving the students in the classroom [4,5]. According to [6], learning for development in the 21st century will eventually become crucial for any type of education system to survive in an age of rapid modern technology. Therefore, the objective of a teacher is to prepare students to be professionals able to successfully find employment. It is not just important to educate students in the fields of modern technologies, models, techniques, and processes, but also to develop their social skills, communication capabilities, competitive nature and the ability to work together in a team, often the case in many job environments [7–9].
A flipped classroom is a learning environment currently being studied worldwide by the teachers. It means that activities usually happening in the class are now taking place outside the class, as well as vice versa [10,11]. Reference [12] argued that while students listen and take notes, the instructor no longer needs to speak for 1 or 2 h, they can use them entirely for discussion and problem-solving with students in-class time. Besides, Al-Rowais et al. [13] has done work on flipped classrooms to look at perception, motivation, commitment, accomplishment, and active learning. Reference [14] reported that the flipped classroom was a reorganization of the classroom environment and home activities. For that reason, the lecturer can reduce the quantity of time spent in the school on lecturing, opening class time for using active learning strategies such as discussion and problem-solving among students in the existence of lecturer [15,16]. However, Reference [17] found that online modules instructed students had higher ethical decisions compared to traditional classes taught students. Reference [11] stated that the findings reveal that the video lecture is the perceptions of the students on pre-classroom learning material that ranks as high as possible. Additionally, the study showed that the pre-classroom learning experience motivates the learning interest of the students as well as strengthens their comprehension of the learning context.
The Universidad Católica de Temuco Center for Teaching Growth and Innovation in Chile has developed a course transformation model that aims to enhance the learning outcomes and attitudes of students towards STEM undergraduate courses. Such courses have historically been marked by high failure rates and students’ low satisfaction levels. The model takes into account students’ learning needs in the context of a competence-based higher education institution and the high level of socio-academic vulnerability of students (housing conditions, family income, University Selection test scores, etc.) [18]. This model is focused on the implementation of learning activities and student-centered teaching, with a specific emphasis on improving problem-solving, study skills, and linking the pedagogical process to real-world situations [16]. Reference [19] have created a flipped classroom teaching assessment framework based on the CIPP model, which can direct educational decision-makers to perform a thorough evaluation of the flipped classroom teaching method from multiple dimensions. Starting in the early twentieth century, work on critical thought in foreign countries began to gain popularity in the 1960s. Educators began exploring the use of critical thinking in teaching in the 1980s and enabled substantial research. The literature indicates that the flipped teaching paradigm offers both opportunities and challenges. To date, however, only a few studies have examined the learning outcomes and difficulties of the flipped classrooms [4].
The impact of modernization is a significant issue for Iraqi society. Everything is evolving, and so the teacher and the method of teaching will have to adapt to keep up with the changes. Some of these technological advances lead to a period in the field of education that requires some revolution. All of this includes the implementation of new teaching strategies focused on various techniques to evaluate and classify for more effective instruction. We, therefore, suggested the Analytical Hierarchy Process (AHP) test teaching strategies, which included conventional learning (face-to-face learning) and flipped learning in the classroom.
The AHP is a concept of measurement for handling with measurable and intangible criteria or attributes that have applied to various areas, such as decision theory and conflict resolution [18,20,21]. Reference [22] stated that the AHP method has widely applied in many fields, and it is a good and great solution for analyzing and solving complex decision problems with multiple criteria. The AHP method is flexible and efficient in the decision-making process. The technique is useful in establishing priorities (relative weight) by decision-makers to each benchmark through pairwise comparison. Also, it is making the most efficient or the best decision when both quantitative and qualitative aspects of a decision need to be considered [23–25]. Also, the AHP technique is particularly easy to utilize when it implemented as decision support software. However, to date, only scant studies have assessed the flipped classroom using multi-criteria decision-making as well as evaluating the flipped classroom by the AHP technique.
This paper presents a study that aims to capture additional aspects of education that supplement student’s expertise and also encourage them to learn new things more efficiently. A model is developed for a different method for teaching science, especially chemistry in Iraqi schools. The rationale behind this research study is to explore in-depth the theoretical foundations, concepts, and definitions of what flipped classroom learning are all around. A sample of secondary school’s boys is selected randomly and divided into two groups for experimental evaluation. All this is to expose, explore, and explain the effect of both types of learning on the level of achievement, motivation, and creative thinking.
This study also examined the potential of applying AHP to the ranking of teaching methods based on expert evaluation under multiple criteria relevant to this field. The criteria weights are obtained via AHP processes to be used then in the ranking of alternatives or in comparing between traditional learning and flipped classroom learning. The AHP includes multiple measures that include setting assessment criteria and their weights, then assessing the flipped classroom methodology compared to the conventional method of learning. This paper demonstrates that in the case of a flipped classroom, all attributes or parameters compare very well with the traditional system. Review of our criteria, which are essential for any active learning, showed that the methods used in the flipped classroom have good potential to expand. The research structure is as follows; the plans are discussed in the next section, including a comprehensive explanation of the AHP approach and supported by results and discussions in Section 3. The last part concludes the study.
We proposed a research model for analyzing a collection of boys’ schools focused on Iraqi society to evaluate the after-effects of flipped classroom learning to assess the students’ creative thinking, motivation, and achievement. The model is designed to analyze the consequences of flipped classroom learning for teaching methodologies in Iraqi schools in order to validate the attitude of the student towards the subject of science, especially in the field of chemistry. Regarding empirical research, the goal was to establish whether and under what circumstances flipped classroom learning was more effective than traditional cognitive learning alone.
A set of 70 students of secondary schools exclusively for boys was selected randomly and divided into two groups equally (each team consists of 35 students), experimental groups and control groups to analyze them. One group is taught with a traditional cognitive approach, while another is prepared using the flipped classroom technique along with some quizzes to promote the competitive environment. Both groups discussed the same material and used the same tests. The learning activities of groups were impacted through different methodologies with a teaching period of two weeks and three lessons per week.
In this respect, we proposed five criteria with their description concerning active learning based on literature and validated by experts from professional practice. Tab. 1 describes the various assessment criteria used to rank the teaching methodologies available.
Table 1: Evaluation criteria with its description
The evaluation of the study is done about five criteria by taking the feedback of six experts in this field involved in the experiment through questionnaires. The gathered data came through two sets of inquiries from the experts. In the first collection, the decision-makers were asked to use simple linguistic terminology, as shown in Tab. 2, to determine the relative importance of the criteria. Whereas, the second collection of the questionnaire is concerned with the ranking of each teaching method (alternatives; traditional and flipped learning) under each approach. In this regard, we employed the AHP technique for the assessment of criteria weights and alternatives rating. A normalized weight is obtained for every teaching method using the AHP technique based on their usage and significance in the teaching methods for all the criteria.
Table 2: Saaty’s nine-point scale of relative importance
Saaty in [26,27] had proposed the AHP technique. It is a robust multi-criteria decision-making approach used to examine complex and unstructured problems in various decision-making circumstances, including, but not limited to, health, agriculture, education, and many application areas [18,28]. The key feature of this method is the use of pairwise comparisons, the comparison of alternatives on the different attributes, and the measurement of weights of the attributes [29]. In a pairwise comparison, this technique uses the judgment of a decision-maker, using the nine-point scale of Saaty as given in Tab. 2. The vast experience and in-depth knowledge of the decision-maker relating to the problem help in a paired comparison [30,31].
AHP is used to assess, rank and select based on a wide variety of alternatives to the decisions, to make a decision making depends on a systematic and numerical process. Typically this method makes it easier to evaluate the relative weights of each criterion compared to the others. The AHP procedure requires four phases.
Decompose the problem into a hierarchy, as displayed in Fig. 1.
Figure 1: Structuring a hierarchy
Gather feedback of trade-off information by paired comparison of elements of the decision. The different criteria that form a single hierarchy can be transformed into a matrix structure to construct a comparison matrix.
Using the term ‘Eigenvalue’ to determine the relative weight of the criteria of the hierarchy. The own principal value of Lambda max 
can be determined in Eq. (1) as follows:
Criteria weights are obtained through a mixture of relative weights to allow an aggregate comparison of all alternatives. As a metric of consistency, the Consistency Index (CI) is derived from the following Eq. (2), where 
is the significant eigenvalue matrix, and n is the number of classes.
To regulate the assessment and scale of consistency analysis [19], The consistency ratio (CR), as described by Eq. (3), is calculated as the consistency measure between the comparative matrix of pairs [27].
where the Ratio Index is at RI for different criteria, is as shown in Tab. 3.
The consistency value should be less than 0.1 [32]; otherwise, the weights of individual themes should be allocated, revalued, and recalculated. The CR values are within acceptable limits for the different layers and their respective groups, and thus imply the degree of consistency in the matrix in a pair type. AHP technique typically has a range of significant advantages. For instance, for decision-makers, this is a reasonably easy way. Also, AHP requires no complex mathematical operations and functions on principles of decomposition, collecting information for pair comparison, and generating and synthesizing priorities for vectors.
3.1 Normalized Weights for Criteria
Once the hierarchical structure has been constructed, the relative contribution of each criterion is obtained through a paired comparison from each decision-maker using the Saaty scale, as shown in Tab. 2. In the AHP procedure, the opinion of an expert is used to obtain final judgments. The decision-makers’ input plays a significant role in getting the comparison pairwise. A single decision-maker is usually assumed to be appropriate to offer a decision in the AHP method. Sole decision-makers, however, can provide an uncertain judgment sometimes. Community group decision-making can be employed to reduce this dubious judgment. In this paper, six decision-makers took part in making a paired comparison based on their experience in teaching methodologies, and this can be a great help in making a biased less decision. After that, six matrices of pairwise comparisons were established for the criteria based on the judgments provided by each expert. Normalized weight for each test as a result of pairwise comparisons from each respondent is displayed in Fig. 2 as six matrices. Each matrix includes the judgments from each expert along with the normalized weight of each criterion. The evaluation is done by calculating the weights and CR by AHP-OS software [33].
Figure 2: The expert’s judgments, along with the normalized weight of every criterion
Fig. 2 illustrates the various evaluation criteria with their contributions used to rank the teaching methodologies available. Normalized weight for each criterion weight is obtained using the AHP technique, as shown in Fig. 3, based on its use and importance in the teaching methods.
Figure 3: The normalized weight of every criterion from each expert
Then the aggregated criteria weights are acquired through averaging the normalized weights to each criterion based on the results from the six matrices using the AHP method. Accordingly, the outcomes of the evaluation have been created, as given in Tab. 4.
Table 4: The final weights of criteria
For all the criteria, the graphic illustration of result processing is gained by using the AHP method, as illustrated in Fig. 4.
Figure 4: Normalized weights of all criteria
The graphic shown above confirmed that the potential for adaptation has the most substantial impact index represented in the graphic as working environment criterion with weight equal to 0.286, followed by Teaching Techniques described in the graphic as a second important criterion with weight the same to 0.23.
After deciding the contributions and weights of the criteria, the ranking of the alternatives is the next step for selecting the best method in teaching methodologies under study. Therefore, each respondent’s decision matrix was defined based on a comparison of the alternatives for each criterion. The decision matrix from each respondent has been established based on comparing the other options concerning every test. Thus, from each expert, five decision matrices were collected and analyzed using the AHP process. The aggregated scores are then obtained by averaging the weights associated with each criterion, as shown in Tab. 5.
Table 5: Averaging the weights of alternatives associated with each criterion
The final ranking of each teaching method is determined by multiplying every weight of criteria with the values of alternatives. Then, add the result of each row together. Tab. 6 and Fig. 5 represent the ranking and scores of the two teaching methods.
Table 6: Ranking of teaching methods
Figure 5: Graphical representation of scores for teaching methodologies
Based on Tab. 6, the alternative at first rank is considered as the best maximization of expected benefits of teaching methods for implementation in the Iraqi schools. The results show that all experts prefer the flipped classroom approach, and it will be the most recommended to the management of the schools.
In conjunction to Tab. 6 and Fig. 5 shows that the slope of changes the method of teaching has increased rapidly from 0.2 to 0.8. The graph also shows that values between 0.6–0.8 is most likely related to flipped learning while all values less than 0.4 are related to traditional learning. At instant, teaching techniques scored 0.83 in case of flipped learning, but it scored 0.17 in traditional learning.
After analyzing our findings, it can be concluded that using flipped classroom learning can have beneficial effects on the output of the student. As a result, our research on a group of Iraqi schoolboys has demonstrated that making them train in the community will dramatically improve their adaptation ability with their teachers’ comparatively less effort.
The flipped classroom, students, has been provided online access to a series of short video lessons that can be completed at their convenience. At the end of each video, a relatively brief online quiz consists of two to four questions designed to test student interest and knowledge of learning. Until moving to the next video, students had to answer correctly by 80%. The purpose of the quizzes is to gain instant feedback, not classification. The class time is devoted to constructive learning sessions, enabling direct contact with the teachers as students apply their learning to solve graded tasks, assessments, and team case studies. Students initially considered the class structure and design somewhat confusing and were initially hesitant to do so. Such kinds of access to the professor were mostly new to students.
Nevertheless, as the students began to see the teacher as a facilitator rather than an educator, they gradually became confident, asking questions for a better understanding of them. The trend of utilizing flipped classroom as a teaching and learning tool is rapidly broadening into education. The correct choice of implementing modern technology will avoid technical problems and obstacles to avoid the pitfall of remote learning and technology. The research tries to suggest a flipped classroom evaluation for better attainments and positive class interaction. The findings of the investigation emphasized several critical issues and criteria for integrating/implementing a flipped classroom technology option. The article can also be specified that the pedagogical effect of teachers and students on teaching and learning are playing a significant role in selecting the appropriate modern learning technology. This study also confirms the significance of having flexible online applications to take advantage of such technology. This research was then implemented in the flipped classroom to ensure that students have become able to effectively use ICT in watching video lectures from anywhere before attending the class.
From the research that has been carried out, it is possible to conclude that the proposed quality assessment strategy has improved by the original AHP technique to establish quality criteria weights. This method is applied in the real circumstances when educational institutions have to decide on the use of specific learning activities for improving their teaching and learning systems. The findings of the present study indicated that the use of a flipped classroom learning method in science classes in Iraqi schools would improve the teaching of chemistry and has affected students’ achievement goals. The research will now reveal some guidelines for supervisors, teachers, and administrators to develop science teaching and overall development of student performance. Consequently, the flipped learning approach should be used by teachers as a new teaching method in Iraqi schools, especially when suitability and possibility can prove to match the characteristics of students. The proposed method is ready to apply forward in practice. Based on the promising findings presented in this paper, work on the remaining issues, e.g., students’ feedback, is continuing and will be presented in future articles.
Funding Statement: The authors received no specific funding for this study.
Conflicts of Interest: The authors declare that they have no conflicts of interest to report regarding the present study.
1 D. C. D. Van Alten, C. Phielix, J. Janssen and L. Kester. (2019). “Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis,” Educational Research Review, vol. 28, pp. 1–18. [Google Scholar]
2 H. J. Mohammed, M. M. Kasim, E. A. Al-Dahneem and A. K. Hamadi. (2019). “Evaluating different multimedia learning methodologies by using the MACBETH method,” in Recent Trends in Applied and Associated Mathematical Sciences. Malaysia: Penerbit Univ Kebangsaan, pp. 58–73. [Google Scholar]
3 G. S. Mason, T. R. Shuman and K. E. Cook. (2013). “Comparing the effectiveness of an inverted classroom to a traditional classroom in an upper-division engineering course,” IEEE Transactions on Education, vol. 56, no. 4, pp. 430–435. [Google Scholar]
4 G. Akçayır and M. Akçayır. (2018). “The flipped classroom: A review of its advantages and challenges,” Computers and Education, vol. 126, pp. 334–345. [Google Scholar]
5 K. R. Clark. (2015). “The effects of the flipped model of instruction on student engagement and performance in the secondary mathematics classroom,” Journal of Educators Online, vol. 12, no. 1, pp. 91–115. [Google Scholar]
6 M. H. Shoon. (2013). “The shift towards an innovation-based model of learning,” in Science and Information Conf., London, UK, pp. 708–711. [Google Scholar]
7 M. Zheng, C. C. Chu, Y. J. Wu and W. Gou. (2018). “The mapping of on-line learning to flipped classroom: Small private online course,” Sustainability, vol. 10, no. 3, pp. 1–14. [Google Scholar]
8 H. J. Mohammed, M. M. Kasim and I. N. Shaharanee. (2017). “Evaluation of e-learning implementation in higher education using multi-criteria technique,” International Journal in IT & Engineering, vol. 5, no. 4, pp. 6–12.
9 H. J. Mohammed, M. M. Kasim, A. K. Hamadi and E. A. Al-Dahneem. (2018). “Evaluating of collaborative and competitive learning using MCDM technique,” Advanced Science Letters, vol. 24, no. 6, pp. 4084–4088. [Google Scholar]
10 K. E. Snowden. (2012). “Teacher perceptions of the flipped classroom: Using video lectures online to replace traditional,” Ph.D. dissertation. University of North Texas, Denton. [Google Scholar]
11 H. J. Mohammed, E. AL-dahneem and A. Hamadi. (2016). “A comparative analysis for adopting an innovative pedagogical approach of flipped teaching for active classroom learning,” Journal of Global Business and Social Entrepreneurship, vol. 3, no. 5, pp. 86–94. [Google Scholar]
12 J. Bergmann and A. Sams. (2012). Flip Your Classroom: Reach Every Student In Every Class Every Day. Eugene, Or., Alexandria, Va.: International Society for Technology in Education, . [Online]. Available: https://cmc.marmot.org/Record/.b41938823. [Google Scholar]
13 A. S. Al-Rowais. (2014). “The impact of flipped learning on achievement and attitudes in higher education,” International Journal for Cross-Disciplinary Subjects in Education, vol. 4, no. 1, pp. 1914–1921. [Google Scholar]
14 J. L. Bishop and M. A. Verleger. (2013). “The flipped classroom: A survey of the research,” in ASEE National Conf. Proc., Atlanta, GA. [Google Scholar]
15 S. Z. Osman, R. Jamaludin and N. E. Mokhtar. (2014). “Flipped classroom and traditional classroom: Lecturer and student perceptions between two learning cultures, a case study at Malaysian polytechnic,” International Education Research, vol. 2, no. 4, pp. 16–25. [Google Scholar]
16 H. Turra, V. Carrasco, C. González, V. Sandoval and S. Yáñez. (2019). “Flipped classroom experiences and their impact on engineering students’ attitudes towards university-level mathematics,” Higher Education Pedagogies, vol. 4, no. 1, pp. 136–155. [Google Scholar]
17 L. McManus, N. Subramaniam and W. James. (2012). “A comparative study of the effect of web-based versus in-class textbook ethics instruction on accounting students’ propensity to whistle-blow,” Journal of Education for Business, vol. 87, no. 6, pp. 333–342. [Google Scholar]
18 T. Dar, N. Rai and A. Bhat. (2020). “Delineation of potential groundwater recharge zones using analytical hierarchy process (AHP),” Geology, Ecology, and Landscapes, vol. 22, no. 3, pp. 1–16. [Google Scholar]
19 J. Xie, T. Zhang and F. L. Cheng. (2017). “Construction of the evaluation system of flipped classroom based on CIPP,” Modern Distance Education Research, vol. 5, pp. 95–103. [Google Scholar]
20 H. J. Mohammed, I. A. M. Al-Jubori and M. M. Kasim. (2019). “Evaluating project management criteria using fuzzy analytic hierarchy Process,” AIP Conference Proceedings, vol. 2138, no. 1, pp. 0400181–0400186. [Google Scholar]
21 S. Hashemi, A. Marzuki, H. J. Mohammed and S. Kiumarsi. (2020). “The effects of perceived conference quality on attendees’ behavioural intentions,” Anatolia, vol. 31, no. 3, pp. 360–375. [Google Scholar]
22 G. Baffoe. (2019). “Exploring the utility of analytic hierarchy process (AHP) in ranking livelihood activities for effective and sustainable rural development interventions in developing countries,” Evaluation and Program Planning, vol. 72, pp. 197–204. [Google Scholar]
23 H. J. Mohammed, M. M. Kasim and I. N. Shaharanee. (2018). “Evaluation of e-learning approaches using AHP-TOPSIS technique,” Journal of Telecommunication, Electronic and Computer Engineering, vol. 10, no. 1–10, pp. 7–10. [Google Scholar]
24 H. J. Mohammed, M. M. Kasim, E. A. AL-Dahneem and A. K. Hamadi. (2016). “An analytical survey on implementing best practices for introducing e-learning programs to students,” Journal of Education and Social Sciences, vol. 5, no. 2, pp. 191–196.
25 P. De Marinis and G. Sali. (2020). “Participatory analytic hierarchy process for resource allocation in agricultural development projects,” Evaluation and Program Planning, vol. 80, 101793. [Google Scholar]
26 T. L. Saaty. (1977). “A scaling method for priorities in hierarchical structures,” Journal of Mathematical Psychology, vol. 15, no. 3, pp. 234–281. [Google Scholar]
27 T. L. Saaty. (1980). The analytic hierarchy process: Planning, priority setting, resources allocation. New York: McGraw-Hill. [Google Scholar]
28 H. J. Mohammed, M. M. Kasim and I. N. Shaharanee. (2017). “Selection of suitable e-learning approach using TOPSIS technique with best ranked criteria weights,” AIP Conference Proceedings, vol. 1905, pp. 0400191–0400196. [Google Scholar]
29 E. Løken. (2007). “Use of multicriteria decision analysis methods for energy planning problems,” Renewable and Sustainable Energy Reviews, vol. 11, no. 7, pp. 1584–1595. [Google Scholar]
30 Q. N. Naveed, M. R. Qureshi, N. Tairan, A. Mohammad, A. Shaikh. (2020). et al., “Evaluating critical success factors in implementing e-learning system using multi-criteria decision-making,” PLoS ONE, vol. 15, no. 5, e0231465. [Google Scholar]
31 R. I. Lucas, M. A. Promentilla, A. Ubando, R. G. Tan, K. Aviso. (2017). et al., “An AHP-based evaluation method for teacher training workshop on information and communication technology,” Evaluation and Program Planning, vol. 63, pp. 93–100. [Google Scholar]
32 J. Malczewski. (1999). GIS and Multicriteria Decision Analysis. John Wiley & Sons, . [Online]. Available: https://www.wiley.com/en-us/GIS+and+Multicriteria+Decision+Analysis-p-9780471329442. [Google Scholar]
33 K. Goepel. (2018). “Implementation of an online software tool for the analytic hierarchy process (AHP-OS),” International Journal of the Analytic Hierarchy Process, vol. 10, no. 3, pp. 469–487. [Google Scholar]
 | This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |