Since problem solving is a crucial component of all mathematical learning, mathematical problem solving skills are an essential component of mathematics [1] Thus, learning mathematics that involves solving mathematical puzzles is considered to be good mathematics learning. One of the objectives of studying mathematics for pupils is to be able to solve problems [2]. Moreover, [3] asserts that the following three reasons highlight the significance of possessing mathematical problem solving skills: (1) problem solving is a fundamental objective of mathematics education; (2) problem solving, encompassing techniques, procedures, and strategies, is a central and primary process in mathematics curricula; and (3) problem solving is a fundamental skill in mathematics acquisition. According to [1], students who possess strong mathematical problem-solving abilities are able to: (1) determine whether the data available for problem-solving is adequate; (2) create mathematical models of common situations or problems and solve them; (3) select and utilize strategies to solve mathematical problems and/or problems outside of mathematics; (4) explain and interpret results in accordance with the original problem and verify that the results or answers are correct; and (5) apply mathematics meaningfully. [4] also made the same point about the significance of problem-solving abilities, namely: (1) use problem-solving to acquire new mathematical knowledge; (2) solve problems that come up in mathematics and other contexts; (3) apply and modify various suitable strategies to solve problems; and (4) keep track of and reflect on the process of solving mathematical problems [4] 

The steps of [3] problem solving model must be followed in order to solve a mathematical problem: (1) comprehend the problem; (2) plan the problem solving process; (3) solve the problem in accordance with the plan; and (4) reevaluate the process and solution outcomes. These phases serve as markers for the evaluation of pupils' aptitude for solving mathematical problems. Thus, a sequence of mathematical operations aimed at resolving an issue can be defined as problem solving abilities. These operations include problem understanding, problem planning, problem application, and re-examining the solution outcomes. According to Sumarmo [5], problem solving skills can be used to verify, develop, or test hypotheses as well as solve story problems, non-routine problems, and apply mathematics in real-world contexts. As a result, there are exercises in problem solving that help students improve their mathematical ability.

However, the Organization for Economic Co-operation Development (OECD) reports that the evaluation results of the Programme International Student Assessment (PISA), which is an organization that conducts studies related to international student assessments held every three years related to change and relationship, space and shape, quantity, and uncertainty and data, show that Indonesian students' mathematical problem-solving skills are still lacking, particularly when it comes to answering non-routine or high-level questions [5].It is also a truth that mathematics proficiency among Indonesian pupils remains low, both in the subject and cognitive domains. The assessment of the cognitive dimension is divided into three domains: (1) knowledge, which includes facts, concepts, and procedures that students must know; (2) application, which focuses on students' ability to apply knowledge and understanding of concepts to solve problems or answer questions; and (3) reasoning, which focuses on solving non-routine problems, complex contexts, and carrying out numerous problem solving steps [6]. PISA model questions are characterized by three primary features: (1) content, which is the subject matter taught in schools and used as the basis for assessment items; (2) context, which is a real-world problem; and (3) competence, which is the capacity to formulate, interpret, and apply mathematics to solve problems [7,8] 

The low PISA scores of Indonesian students demonstrate that the country's students struggle to comprehend ideas and apply what they learn to solve issues in practical settings. Even though PISA questions need students to have mathematical problem-solving skills, low mathematical problem-solving skills have an impact on students' incapacity to solve problems with features like PISA questions [5]. According to the results of the 2nd Mathematics Literacy Contest (KLM), which was held in 2011 for junior high school students throughout South Sumatra by the Sriwijaya University Postgraduate Mathematics Education Study Program, students' ability to solve problems whose context is very different from what they typically encounter at school is still very low; in fact, students only seem to pay attention to the pictures without understanding what needs to be done [9].Poor problem-solving abilities among students [10,2] It is also stated that one of the issues that teachers encounter most frequently is the absence of specifically created questions that align with the abilities and personalities of their students. 

A similar incident occurred in SMP Negeri 1 Kolaka. According to the findings of teacher interviews focused on mathematics, non-routine problems—particularly those that help pupils hone their mathematical problem-solving abilities—are still infrequently assigned by math teachers. In order to avoid developing a habit of working on complex issues, the growth of mathematical power is not put into practice. This is corroborated by the findings of the examination of the difficulties that instructors frequently assign to their pupils, which show that the majority of these problems are regular or everyday concerns. On the other hand, presenting difficult tasks might spur pupils' interest in solving the provided mathematical puzzles. This aligns with the perspective of Sugiman [11] 

That kids' motivation to solve mathematical problems can be sparked by tasks that are just a little bit challenging but yet fall within the Zone of Proximal Development (ZPD). Therefore, in an attempt to encourage students to improve their problem solving skills, PISA-based mathematical problem solving skills questions must be developed. Higher order thinking capabilities in mathematical problems have been shown in several studies to encourage pupils' problem solving abilities. According to Lewy [12] the creation of higher order thinking skills questions may have an impact on students' capacity for logical, critical, and creative thought as well as their ability to solve mathematical problems. Furthermore, non-routine questioning has been shown to improve learning at the elementary, junior high, and high school levels [12]

Many studies have been done on the creation of test instruments for students' mathematical problem solving abilities. These include [13] on the creation of test instruments for students' mathematical problem solving abilities based on solo taxonomy; [14] on the creation of test instruments for students' mathematical problem solving abilities; and [5] on the creation of test instruments for students' mathematical problem solving abilities based on change and relationship content. While Pisa-based mathematical problem solving ability test instruments contain all Pisa content—change and relationship, space and shape, amount, and uncertainty and data—some research have not addressed and developed them adequately. Consequently, this study primarily examines and proposes a Pisa-based student mathematical problem solving ability test instrument and its assessment in light of the urgent need for such a tool and the lack of growth in this area. 

Research and development (R&D) is the focus of this study. Tessmer's formative assessment development approach is the one that is applied. Formative evaluation is a step in the problem discovery process of product design and development, according to [15] This study was conducted in two phases: the formative assessment phase, which involved design and preparation, and the preliminary preparation stage [16] Self-assessment, expert evaluations, one-on-one and small-group discussions, and field testing are all included in the formative evaluation stage [15] 

Determining the location and topic of the study, question analysis, interviews, and observation are all done during the preparatory phase. The PISA model's grids and math questions were created throughout the design phase using the problem-solving skill indicators as a guide. During the self-evaluation phase, the questions that were created during the design phase are examined. Prototype I was the end product. During the prototype phase, a number of tasks were completed, including: (1) expert review, which involved having experts evaluate the questions' validity; and (2) one-to-one testing, which involved testing the questions on three students who were not the subjects. Prototype II will be created when the trial's results are amended; prototype III will be created in a small group setting using six non-subject students; and prototype IV will be produced in a field test using the study subject. Students at SMP Negeri 1 Kolaka in grade VIII served as the study's subjects. 

A five-scale Likert scale was used to analyze student response questionnaires to gather data on the practicality of the questions. This data analysis was divided into two categories: (1) validity, which includes data on validity, reliability, question difficulty level, and question differentiation; and (2) practicality. [17,18] state that the following formula can be used to determine the average practicality score: 

The categorisation of practicality refers to the criteria for instrument practicality in Table 1.

Table 1. Instrument Practicality Criteria

The goal of developing this PISA-based exam instrument is to provide a reliable, useful, and efficient tool that can assess students' problem-solving skills and place them according to either PISA or international norms. Two phases were used to carry out this development: the formative evaluation stage and the preliminary preparation stage.

Preparation Stage (preliminary) 

Gathering a number of sources relevant to the study is the first step in this process. In addition, the investigator carried out tasks to ascertain the location and topic of the study. SMP Negeri 1 Kolaka served as the study's test location, and the participants were the school's VIII-grade pupils. The researcher then created the question set, which entailed creating the PISA model's grids and math problems using the previously acquired literature as a guide. The PISA 2012 Draft Mathematics Framework was used to determine the content components for this stage, which included Change and Relationships, Space and Shape, Numbers, Probability/Uncertainty, and Data. Additionally, every question is representative of every PISA level. The questions were written as descriptions, using language that was appropriate and according to the Enhanced Spelling.

Formative Evaluation Stages

Self evaluation

At this point, the six-item self-evaluation was taken from prototype I, a mathematical problem in the PISA model designed to gauge pupils' aptitude for solving mathematical problems. At this point, researchers went over the questions that had been created during the design phase to identify and fix any mistakes or shortcomings in the first prototype that had been created to satisfy the requirements for language, build, and content.

Expert Review

Two experts conducted the validity test or expert review phase. A validation sheet providing open validation along with helpful comments yielded the validity results. The PISA model mathematics question instrument was rated as satisfactory (valid in terms of content and language) by all validators, and it may be used to gauge students' problem-solving abilities based on PISA.

One-to-one

Three eighth-grade SMP Negeri 1 Kolaka pupils were tested as non-subjects at this point. The three ability-leveling students—high, medium, and low—worked on six PISA model mathematics tasks. After that, the researcher engaged in conversation and interaction with these pupils to find out what challenges they were having with the PISA model math problems. Most students were able to comprehend the challenges at the one-to-one level. Students with high and medium ability levels demonstrated strong problem-solving skills in mathematics. While children with limited ability received low scores for their ability to solve mathematical problems. Although low ability pupils need a long time to learn, most students can read problems, understand problems into mathematical problems, and plan solutions. This demonstrates that, although occasionally being unable to solve problems with a high cognitive level, students at the one-on-one stage generally have good mathematical problem solving skills. After expert assessments and one-on-one consultations, the questions in prototype I were revised to create prototype II, which consists of six questions that will be evaluated in small groups.

Small Group

Following expert evaluations and one-on-one testing, six non-subject VIII grade students from SMP Negeri 1 Kolaka were tested on the amended questions in small groups. There were two high ability pupils, two medium ability students, and two low ability students among the students. Overall, the responses given by students in the small group and the ones given by them one-on-one were nearly identical. Even so, there were still certain difficulties that pupils were unable to solve and some mistakes in their solutions. In this small group setting, students may answer basic calculation problems with ease. The researchers watched students as they worked on the assigned issues in small groups. In contrast to the one-on-one test, students' level of questioning regarding the questions' meaning had dropped throughout the small group testing phase. The small group stage is also utilized to evaluate the design of the question instrument in terms of its practicality, degree of difficulty, and dependability.

1. Reliability

In order to establish the instrument's trustworthiness as a measuring tool, reliability testing aims to ascertain the instrument's consistency. If the value is more than 0.6, the calculation's conclusions, as determined by the Cronbach alpha reliability test formula, can be considered reliable. Furthermore, reliability data = 0.95 was obtained from the Cronbach alpha test results, allowing the question to be deemed reliable.

2. Difficulty Level

The following results of the analysis of the level of difficulty of the PISA model question items to measure students' mathematical problem solving skills can be seen in table 2 below:

Table 2. Level of Difficulty of Small Group Trial Results.

Based on Table 2, it is known that in the trial question number 1 has a level of difficulty in the ‘Easy’ category, question numbers 3, 4, 5 and 6 have a level of difficulty in the ‘Medium’ category, for question number 2 has a level of difficulty in the ‘Difficult’ category.

3. Distinguishing Power

The results of the Distinguishing Power analysis of the test instrument items can be shown in table 3 below:

Table3. Distinguishing Power of Small Group Trial Results

Based on Table 3, it is known that in the small group trial all questions have sufficient differentiating power.

4. Consider the practicality 

The findings of students' answers to the PISA-based problem solving abilities questions were used to gather information about the topic's practicality. After the response data was certified and analyzed, the practicality value (V ̅) of the question was determined to be 3.89. The designed instrument falls within the practical category based on the qualifications.Additionally, prototype III, which consists of six questions, was kept and tested in the field test based on the findings of the item analysis and the remarks and suggestions made by the students in this small group. 

Field Examination 

The purpose of this field test is to ascertain whether or not PISA model mathematical problem-solving questions have an impact on students' ability to solve mathematical problems. These field test questions have been proven to be legitimate and useful. Following a validation procedure in which multiple validators provided comments and suggestions for enhancing the questions' language, construct, and content, the questions were deemed valid. Using the scoring rules as a guide, the average mathematical problem solving ability of pupils was calculated by analyzing data from the PISA test results. Table 4 below provides a description of students' proficiency in solving mathematical problems. 

Table 4. Distribution of Students' Mathematical Problem Solving Ability

13 students (46.43%) have very high mathematical problem solving ability, 9 students (32.14%) have high mathematical problem solving ability, 2 students (7.14%) have moderate mathematical problem solving ability, 2 students (7.14%) have low mathematical problem solving ability, and 1 student (7.14%) has very low mathematical problem solving ability, according to the above table.The average score for pupils' ability to solve mathematical problems is 71.32, which puts them in the high category, according to the data analysis results. Furthermore, an analysis of the student replies was conducted, yielding an average student response of 57.17, indicating a favorable reaction based on the criteria. These findings are consistent with earlier studies that found pupils responded favorably to the creation of PISA-based arithmetic problems [19]. As a result, the PISA question bank designed to gauge students' aptitude for solving mathematical problems satisfies the necessary requirements.

It may be inferred from the research findings and the discussed discussion that the PISA-based test of mathematical problem solving ability satisfies the requirements for a good test instrument, which include validity, reliability, a good degree of difficulty, and differentiation. Additionally, the test instrument designed satisfies the practical and effective criteria.

The authors declare that they have no conflict of interest

No funding sources

The study was approved by the Kolaka University.

1. Ariawan, R., & Nufus, H. Hubungan Kemampuan Pemecahan Masalah Matematis dengan Kemampuan Komunikasi Matematis Siswa. Jurnal THEOREMS (The Original Research of Mathematics), vol. 1, no. 2, 2017, pp. 82–91. https://doi.org/10.31949/th.v1i2.384

2. Ramlan, A. M., Hermayani, & Jahring. Analisis Kemampuan Pemecahan Masalah Matematis Ditinjau Dari Kepercayaan Diri. AKSIOMA: Jurnal Program Studi Pendidikan Matematika, vol. 10, no. 4, 2021, pp. 2188–2199. https://doi.org/10.24127/ajpm.v10i4.3996

3. Sumartini, T. S. Peningkatan Kemampuan Pemecahan Masalah Matematis Siswa Melalui Pembelajaran Berbasis Masalah. Jurnal Pendidikan Matematika STKIP Garut, vol. 5, no. 2, 2016, pp. 148–158. https://doi.org/10.31980/mosharafa.v5i2.270

4. Putri, D., & Warmi, A. Kemampuan Pemecahan Masalah Matematis Siswa dalam menyelesaikan Soal PISA Matematika pada Konten Bilangan. Jurnal Penelitian Pembelajaran Matematika, vol. 15, no. 1, 2022, pp. 138–152. https://doi.org/10.30870/jppm.v15i1.14102

5. Amalia, W., Mulyono, & Napitupulu, E. Pengembangan Soal Matematika Pisa-Like pada Konten Change and Relationship untuk Mengukur Kemampuan Pemecahan Masalah. Paradikma Jurnal Pendidikan Matematika, vol. 11, no. 2, 2018, pp. 1–8. https://doi.org/10.24114/paradikma.v11i3.22925

6. Widyaningsih, E., & Syarifuddin, H. Pengaruh Model Pembelajaran Discovery Learning Terhadap Kemampuan Penalaran Matematis Peserta Didik Kelas VIII SMPN 3 Pariaman. JEPM: Jurnal Edukasi Dan Penelitian Matematika, vol. 9, no. 1, 2020, pp. 50–55.

7. Mashuri, S., Kurniawan, R., & Jahring. Kemampuan Literasi Matematis Siswa dalam Menyelesaikan Soal Berorientasi PISA Konten Quantity. EDU-MAT: Jurnal Pendidikan Matematika, vol. 11, no. 1, 2023, pp. 179–189. https://doi.org/10.20527/edumat.v11i1.15417

8. Pratiwi, A. A., Effendi, M. M., & Ummah, S. K. Pengembangan Instrumen Evaluasi Pembelajaran Matematika Tipe PISA Berkarakteristik Kebudayaan Lokal. Jurnal Pendidikan Matematika RAFA, vol. 6, no. 1, 2020 39–53. https://doi.org/10.19109/jpmrafa.v6i1.4985

9. Jurnaidi, J., & Zulkardi, Z. Pengembangan Soal Model Pisa Pada Konten Change and Relationship Untuk Mengetahui Kemampuan Penalaran Matematis Siswa Sekolah Menengah Pertama. Jurnal Pendidikan Matematika, vol. 7, no. 2, 2013, pp. 37–54. https://doi.org/10.22342/jpm.8.1.1860.25-42

10. Jahring, J., & Marniati, M. Efektivitas Model Pembelajaran Inside Outside Circle (IOC) Terhadap Hasil Belajar Matematika Siswa SMP. Jurnal Karya Pendidikan Matematika, vol. 7, no. 1, 2020, pp. 22–26. https://doi.org/10.26714/jkpm.7.1.2020.22-26 

11. Sunendar, A. Pembelajaran Matematika dengan Pemecahan Masalah. Jurnal THEOREMS: The Original Research of Mathematics, vol. 2, no. 1, 2017, pp. 86–93. https://doi.org/10.31949/th.v2i1.577

12. Hartatiana. Development of Argument-Based Problem Solving Questions for Grade V Students at SD Negeri 79 Palembang. JPM: Journal of Mathematics Education, vol. 8, no. 2, 2014, pp.  76–85. https://doi.org/10.22342/jpm.8.2.1912.76-85

13. Arif, N., Yuanita, P., & Maimunah. Pengembangan Instrumen Tes Kemampuan Pemecahan Masalah Matematis Berbasis Taksonomi SOLO pada Materi Barisan dan Deret. Jurnal Cendekia: Jurnal Pendidikan, vol. 06, no. 02, 2022, pp. 2318–2335. https://doi.org/10.31004/cendekia.v6i2.1498

14. Fitrianty, F., Yunita, A., & Juwita, R. Pengembangan Instrumen Tes Kemampuan Pemecahan Masalah Matematika di SMP Negeri 12 Padang. Lattice Journal : Journal of Mathematics Education and Applied,vol. 2, no. 1, 2022, pp. 91–102. https://doi.org/10.30983/lattice.v2i1.5337

15. Tessmer, M. Planning and Conducting Formative Evaluations: Improving the Quality of Education and Training. Routledge Falmer Taylor & Francis Group, 1993. https://doi.org/10.4324/9780203061978

16. Nasution, R. S., Fauzi, K. M. A., & Syahputra, E. Pengembangan Soal Matematika Model PISA pada Konten Space and Shape untuk Mengukur Kemampuan Penalaran Matematis. Paradikma Jurnal Pendidikan Matematika, vol. 13, no. 1, 2020, pp. 1–10. https://doi.org/10.24114/paradikma.v13i1.22942

17. Jahring, J., Herlina, H., Nasruddin, N., & Astrinasari, A. Pengembangan Instrumen Evaluasi Pembelajaran Matematika Berbasis Online Menggunakan Aplikasi Quizizz. Aksioma: Jurnal Program Studi Pendidikan Matematika, vol. 11, no. 2, 2022, pp. 872–881. https://doi.org/10.24127/ajpm.v11i2.4932 

18. Purwasi, L. A., & Fitriyana, N. Pengembangan Lembar Kerja Peserta Didik (Lkpd) Berbasis Higher Order Thinking Skill (Hots). AKSIOMA: Jurnal Program Studi Pendidikan Matematika, vol. 9, no. 4, 2020, pp. 894–908. https://doi.org/10.24127/ajpm.v9i4.3172

19. Wati, Y. I. Pengembangan Instrumen Soal Matematika Berbasis PISA 2021 (Doctoral dissertation, Muhammadiyah University, Semarang), 2021.