Development of an Item Bank for Measuring Students ’ Conceptual Understanding of Real Functions

ABSTRACT


INTRODUCTION
Major challenge for mathematics education researchers (e.g., Bisson et al., 2016;Code et al., 2014;Crooks & Alibali, 2014) is measuring conceptual understanding with acceptable validity and reliability.To achieve increased conceptual understanding in classrooms there need to be valid and reliable measures of conceptual understanding (Bisson et al., 2016).Test validation is the procedure by which evidence is gathered to determine if the test items satisfactorily represent a concept domain and whether the test measures the properties that it proposes to measure (Day & Bonn, 2011).Reliability indicates to whether a test is consistent

Research Article
within itself and across time and can be measured by using statistical calculations that focus on individual items and the test as a whole (Day & Bonn, 2011).By using a valid and reliable measuring instrument, teachers can see the misconceptions that students still hold about the observed concept and help them overcome these misconceptions.In mathematics education research, students' conceptions are typically explored through oral interviews or written surveys.For assessing the conceptions of a larger number of students (e.g., in a university course), written surveys are more efficient (Mešić et al., 2019).Some authors suggest that it is better to develop comprehensive item banks that can be used for tailoring assessment instruments in order to meet different assessment goals then developing a simple assessment instrument (Bjorner et al., 2007).
The theory of function is of vital importance for learning mathematics (Elia & Spyrou, 2006).Since many phenomena and processes from different areas of the economy and everyday environment are described by functions, it is necessary to examine whether students understand the concepts of functions.A significant indicator of conceptual understanding is ability to represent mathematical situations in different ways and knowing how useful can be different representations for different purposes (National Research Council, 2001).Understanding the concept of function or any other concept entails the ability to recognize at least two different representations of the concept and transition coherently from one to another (Hitt, 1998).Because of the importance of different representations in conceptual understanding, our research focused on three fundamental ways of representing functions: graphical, algebraic, and verbal representation.Although there was some highly valuable research in the field of students' misconceptions in the area of real functions and its properties (Bardele & Ferrari, 2011;Bardini et al., 2014;Bezuidenhout, 2001;Carlson et al., 2010;Dreyfus & Eisenberg, 1982;Elia & Spyrou, 2006;Even, 1998;Gagatsis & Shiakalli, 2004;Habre & Abboud, 2006;Hitt, 1998;Lloyd et al., 2010;Nitsch et al., 2015;Sierpinska, 1992), to the authors' knowledge at this moment there is no real function item bank that would allow for assessing university and high school students' understanding of these concepts related to the real function: function recognition, function's zeros, sign of a function, function evenness/oddness, the limit and asymptotes of a function, the extreme and flow of a function, function convexity/concavity and inflection point.
In this paper we therefore describe the development of conceptual understandings of real functions (CURF) item bank that includes all the above concepts related to the real function and demonstrate its potential for measuring university students' conceptual understanding.We authored and revised multiplechoice items that require manipulation of different representation of the same concept and ability to make a transition from one representation to another.We finally administered these items in a field study and used statistical tests focusing on both item analysis and on the entire test to report on the evidence collected in support of its validity and reliability.Created assessment tool with verified validity and reliability, provides a possibility to measure understanding of concepts across education institutions and over time in a calibrated manner.
Our research questions are: 1. Is the CURF item bank valid and reliable for measuring CURF in introductory mathematics courses at universities?
2. What concepts do students understand and what misconceptions do they have in the area of real function?
3. In which of the transitions between the two representations of the same concept related to the real function are they better?

Understanding the Concepts of a Real Function
In many curricula, the concept of function connects algebra, trigonometry and geometry (Dreyfus & Eisenberg, 1982).The concept of function is complex due to several factors (Dreyfus & Eisenberg, 1982), such as: • It is not a separate concept, but it is connected with a significant number of sub-concepts (e.g., domain, pre-image, variable, extremes, and growth.We can call them 'functional concepts').
• The concept of function can be used to connect unrelated subjects, such as geometry and algebra.This activity is part of the process of abstraction achieved by using functions.
• The same function may be represented in several settings (e.g., as a table, arrow diagram, graph, formula, or verbal description).
Even/odd functions are essential in many areas of mathematical analysis.They are related to the concepts of symmetry, which most often appears in the visual context (Zazkis, 2014).Bezuidenhout's (2001) research showed that first-year students poorly understand the relationship between the concept of limit, continuity, and differentiation.Research by Tall and Vinner (1981) has shown that students have problems adopting the concept of function limits.Students are not usually allowed to see how the concepts of asymptotes and limit are interrelated (Hornsby & Cole, 1986).Also, previous research has shown that students have problems adopting the concept of function derivation (Orton, 1983).Research by Habre and Abboud (2006) on the conceptual understanding of the function and its derivation showed that students tend to use the first formula and then graphics.Bardelle and Ferrari (2011) examined students' understanding of function monotony.The results showed that freshman students have a poor understanding of the standard definition of increasing function and difficulties in application.The authors are not familiar with previous research on understanding other concepts of the real function of one real variable.

Conceptual Understanding and Multiple Representations of a Real Function
Researchers in mathematics education distinguish between conceptual and procedural understanding in mathematics (e.g., Hiebert & Lefevre, 1986;Schneider and Stern, 2010;Skemp, 1976).Conceptual knowledge is a knowledge of concepts (Rittle-Johnson & Schneider, 2015).In the past, conceptual knowledge has been defined as knowledge rich in connections (e.g., Hiebert, 2013), but recently the richness of connections is viewed as a feature of conceptual knowledge that increases with expertise (Rittle-Johnson & Schneider, 2015).Knowledge of procedures-series of steps and actions done to accomplish a goal is called procedural knowledge (Rittle-Johnson & Schneider, 2015).Knowledge of mathematics includes both conceptual and procedural knowledge.An individual is not competent in mathematics if it is lacking in either kind of knowledge (Hiebert, 2013).
Formal mathematics instruction seems to do a better job of teaching procedures than concepts or relationships between them (Hiebert, 2013).This makes sense, because as Sfard (1991) observes, historically, the operational aspect (procedural knowledge) preceded the structural aspect (conceptual knowledge) and Sfard (1991) argues that the same should be the case for the individual learning process, because the structural approach is more abstract than the operational.However, in schools students often learn only the procedures, without truly understanding the concepts behind them (Even, 1990).This happens due to the lack of meaningful learning, which is connected with conceptual knowledge, while procedures may or may not be learned meaningfully (Hiebert, 2013).Often procedures are learned by heart (rote learning), which means they are tied closely to the context in which they were learned, and this knowledge can only be applied in contexts that look a lot like the original (Hiebert, 2013).
Due to the prevalence of procedural knowledge and sadly rote learning in our schools, we wanted to develop the test items bank that would assess conceptual knowledge of real functions (or more precisely, one important aspect of conceptual knowledge-recognition and transition between different representations of concepts related to functions).
The ability to recognize and represent the same concept differently, the flexibility to move from one representation to another, provides insight into relationships, develops deeper and enhanced conceptual understanding, and strengthens problem-solving skills (Even, 1998).A central goal of mathematics teaching is thus taken to be enabling students to move from one representation to another without falling into contradictions (Hitt, 1998).Principles and standards for school mathematics (NCTM, 2000) include standards that relate exclusively to representations and emphasize using multiple representations in learning mathematics.The common core mathematics curriculum based on learning outcomes (APOSO, 2015) within algebra contains the component algebraic expressions, functions, proportions, and applications.The learning outcome includes analyzing and displaying mathematical situations and structures using algebraic symbols and different notations, graphs and diagrams, and making generalizations.Primary mathematical forms of representation and transition play a central role in subject-related didactics (Nitsch et al., 2015).Different mathematical representations of functional dependence and transitions between them have proven critical factors for successful individual learning (Nitsch et al., 2015).Gagatsis and Shiakalli (2004) examined the transitional abilities of university students when it comes to the concept of function-the verbal, graphic and algebraic representation of that concept.Students understand two representations of the same concept (verbal and graphic) as different concepts rather than as different ways of representing the same concept (Gagatsis & Shiakalli, 2004).This fact indicates that they do not have a complete and coherent cognitive structure for the concept of function (Gagatsis & Shiakalli, 2004).Habre and Abboud (2006) concluded that for most calculus 1 course students, algebraic representation dominates when they think about function.Kalchman and Koedinger (2005) noticed that some students lack coordinated conceptual understanding of functions and how they appear in different representations (graphical, tabular, and symbolic representation), and they designed a change in mathematics curriculum, which would teach students to reason about multiple representations of mathematical functions.
In this research, we have concentrated on the verbal, algebraic and graphical representation of content.By verbal representation we mean that the concept is described in written words in the way that is most accessible to students and as described in words in the textbooks.Algebraic representation involves the use of mathematical symbols and representation through mathematical expressions or formulas.The graphical representation of the function represents the geometric representation of the function in the coordinate system.Sierpinska (1992) states that students have difficulties connecting different representations of the concept of function (formulas, graphs, diagrams, and verbal descriptions) when interpreting function graphs and manipulating symbols related to the function.Some authors associate these difficulties with the way concepts are represented in schools.The teacher who understands the concept of function as an action emphasizes function as a chain of operations (Sánchez & Llinares, 2003).Such teachers put more emphasis on algebraic representations and arithmetic activities in teaching.A teacher's goals are determined by different aspects of the featured concept, its specific connections, and how representations are used (Sánchez & Llinares, 2003).

RESEARCH METHODOLOGY Sample and Sample Size
The student sample was obtained by convenience sampling and it included altogether 80 freshman students of the Faculty of Science of the University of Sarajevo: 39 students from the Department of Information Technology (IT), 25 students from the Department of Theoretical Computer Science (TCS), and 16 students from the Department of Chemistry.33 students were female and 47 were male.All students attended lectures on real functions of a real variable-examining and drawing graphs of functions in the final grade of high school ten months earlier.We conducted our research on a sample of first-year university students because previous research has shown that the performance in elementary mathematics of first-year university students is comparable to the performance of high school students.

Development of an Item Bank
The development of an item bank involved several stages, which mainly follow the procedure proposed by Crocker and Algina (1986, as cited in Liu, 2010): (i) defining the construct, (ii) delineating the construct into items, (iii) item survey, (iv) final field testing, and (v) item and test analysis.
The item bank CURF covers the area of real functions of one real variable-examination and graphing of functions, which is studied in the final grade of high school (grammar and technical school) and introductory mathematics courses at universities.Based on the learning outcomes, our experience teaching high school and university mathematics, a review of school textbooks, resources and materials from practical and theory classes at the faculty, we defined the construct as a cross-section of the content being studied at university and high school.We focused on the concepts related to functions presented in Table 1.

Delineating Constructs into Items
We created items that require the ability to transition from source to target representations of the given concept.We divided 36 items into four cards (each card containing nine items).The cards were organized in the following way: • Question card A-items require a transition from graphic to verbal representation (the source representation is graphic, and the target representation of the same concept is verbal).
• Question card B-items require a transition from graphic (source) to algebraic representation (target).
• Question card C-items require a transition from algebraic (source) to graphic representation (target).
• Question card D-items require a transition from algebraic (source) to verbal representation (target).
The English version of the question cards is provided at the following address: https://sites.google.com/view/supplementalmaterialaphd.Most of the items are developed by the authors except items B1 (Bardini et al., 2014) and B6 (Szydlik, 2000).Each item has four offered answers (alternatives), with one correct answer and three distractors (incorrect alternatives).In order to get an insight into cognitive processes, students were expected to explain each chosen answer.During the creation of distractors, we chose situations for which we assumed would trigger student misconception.As an example of representing one concept through cards, we present the concept of the sign of a function trough

Test Validation
The content validation procedure was conducted by questioning five professors from the Faculty of Science at the University of Sarajevo.The professors had to answer three questions for each item:  The questionnaire results showed that in 86% of all cases, the professors agreed that the test item measures the aspect of understanding, and in 87% that it measures the ability to transition from one representation to another.The average difficulty rate of the items was 2.01 on a scale from 1 to 3. In their opinion, the test is of average difficulty level.For card A, on average, the item difficulty was estimated to be 1.88.For card B, it was 2.07; for card C, it was 2.02; and for card D, it was 2.07.Based on these results, we can conclude that card A is slightly more accessible than the other cards.Professors gave suggestions for improving some items (Table 2), which we accepted and applied on the items.

Final Field-Testing
The test was conducted in the classroom under the strict supervision of researchers.Through a group interview, we first checked whether the students had learned the concepts related to the function in the final grade of high school.Those students who did not learn about it were excluded at the beginning of the testing.Students voluntarily participated in the research.In the beginning, we emphasized that each question has four answers offered, out of which only one is correct.Students were required to give a written explanation for each of their chosen answer.At this stage we wanted to check wording and clarity of items.For each of the 36 items, students were asked several questions with the purpose to gain additional insight into technical characteristics of items and their cognitive validity (Table 3).Some questions in Table 3 are original, while others are similar to questions in previous research (Mešić et al., 2019).Since this test will also serve as a mock test which will be the base for the test for measuring the understanding of high school graduate students, Table 3 contains the following question: Do you consider the question appropriate for grammar school students in their final year?

Item Analysis
In order to determine which items in the pool of potential test items will be the best to construct the most efficient, reliable, and valid test we conducted an item analysis.After final field-testing of the students, we calculated descriptive statistics: item difficulty index (p), item discrimination index (D), and point-biserial correlation coefficient (rpbis).Item difficulty index (p) presents the percentage of students who got the correct answer compared to the total number of students.A value of p closer to 1 indicates a more accessible item, while a value closer to 0 indicates a more difficult one.The value of the item difficulty index between 0.3 and 0.9 is considered acceptable (Doran, 1980).
In order to determine how well each item in a test distinguishes between higher-achieving and lowerachieving students, according to their test scores, we calculated item discrimination index (D).Ebel and Frisbie (1991) classify items according to the following rules: D>0.4 (excellent discrimination), D between 0.30 and 0.39 (good discrimination), D between 0.20 and 0.29 (acceptable discrimination), while items lower than 0.2 should be excluded or revised.When we want to discover if an item is good, we determine the correlations between the success of students on the item and their success on the whole test (item-total correlations) (Husremović, 2016).For this purpose, we used a point-biserial correlation coefficient (rpbis).The value of the point-biserial correlation coefficient ranges between -1 and 1.We can consider an item to be good if it was correctly solved by students who did well in the entire test or vice versa.The point-biserial correlation coefficient is considered acceptable if rpbis≥0.2 (Kline, 1986).

Test Analysis
In order to determine test reliability, we calculated Cronbach's alpha, which is the most widely used objective measure of reliability in the case of only one test.George and Mallery (2003) provide the following rules: α>0.9 (excellent reliability), α>0.8 (good), α>0.7 (acceptable), α>0.6 (questionable), α>0.5 (poor), and α<0.5 (unacceptable).In order to measure the discriminatory power of the entire test, we used Ferguson's delta (δ).The discriminatory power refers to the potential of the test to differentiate among the examinees Insert a graph of a function that is neither even nor odd.

A5
In alternatives add "only on the interval…" A6 In alternatives a and c use the plural.

A8
Replace points F and A in alternative d.

B4
Change alternative d to the (−) ≠ −() B5 In alternatives a and b add "only".

B9
Rephrase the question, improve the visibility of the graph.

C7
Include only the property of the growing function, and omit the first derivative.D6 In the alternative b add "vertical, oblique and horizontal" D8 Change the alternative b into: "The maximum of the function is at the point (0, 0)." according to a specific characteristic (Husremović, 2016).If the test has a Ferguson's delta greater than 0.90, it is considered to provide a good discrimination, and it is a good test (Kline, 1986).Calculation of Ferguson's delta relies on the relationship between the overall test scores of any two students (Day & Bonn, 2011).

Questionnaire Results
Results from the questionnaire conducted with university professors showed that most of the items measure conceptual understandings and the ability of transition from one representation to another.Professors consider the test to be of moderate severity.Based on written student surveys we found that students understand the questions.For most students, the test is well-formulated and clear.Students rated most of the items as not complex or moderately difficult.Also, the questionnaire results showed that this test does not represent a cognitive load for students.96% of students claimed no words they did not understand, while the others stated that they were not familiar with words such as 'even', 'odd' and 'inflection point'.For item A2, two students stated that the graph of the function was confusing.Two students reported that they were not familiar with the concept of asymptotes and that the coordinate system was confusing for them.Two students were not familiar with the concepts of convexity and concavity.For item B4, one student stated that answers 'c' and 'd' were confusing.In item B6, the graph was confusing for three students.One student stated that the answers offered in item B9 were confusing.For most items (80.6%), students reported that they needed minimal effort to solve them.19.4% of items were estimated to have an effort level of 3. Students' opinions about the level of difficulty of the items are shown in Figure 5.

Results of Item and Test Analysis
In order to address the first research question, we analyzed items and the test to determine which items in the pool of potential test items will be the best to construct the most efficient, reliable, and valid test.For item and test statistics we used data from final field-testing of the students.We used SPSS 25 software for data processing and coded the students' answers with numbers 0 (incorrect answer), 1 (correct answer) and 9 (missing answer).
The difficulty indices p for each item are shown in Table 4.The average value of the item difficulty index is 0.49, which brings it very close to the optimal of 0.5.Items with a difficulty index value outside the recommended range (0.3-0.9) are items A9, B4, B6, B9, C9, and D8 (difficult items) and D1 (accessible item).We can conclude that 29 (90.6%)items have an acceptable difficulty index.4, 26 items have good or excellent discrimination index (D≥0.3).Lower discrimination index values indicate that there is no difference between higher-achieving and lower-achieving students.It implies that the item is not well formulated or is too difficult or too easy (Ding & Beichner, 2009).Item B5 (Figure 2) has the highest discrimination index value (0.85), so we can state that this item is the best formulated and defined item because it distinguishes well between higher-achieving and lower-achieving students.Item D8 has the worst discrimination index value (D=-0.2).If we consider its difficulty index value, we can conclude that almost all students solve it incorrectly as it belongs to the group of complex items.The average discrimination index value is D=0.44, which represents excellent discriminatory power.
Based on the rpbis coefficient from Table 4, we can conclude that most items in terms of quality belong to the category of very good items.Only four items (A9, C9, D8, and D9) have a point-biserial coefficient rpbis<0.2,so they belong to the category of bad items and have an unacceptable correlation coefficient.Such items should be excluded or revised.The average point-biserial coefficient is 0.35 (acceptable).
In order to produce the trustworthiness of the test, we used Cronbach's alpha coefficient to determine the internal consistency of the item scale.The initial Cronbach's alpha coefficient for the whole scale is 0.788, which shows that the test has good reliability.The discriminatory power of the test is determined using Ferguson's delta.Ferguson's delta is δ=0.9716, which shows that our test is good and has good discriminatory power.

Results of Distractor Analysis
In order to get an insight into the quality of the distractor, we conducted a distractor analysis.Through distractor analysis we addressed the second research question, since the choice of a distractor as the correct answer is an indicator of students' difficulties and misconceptions.Distractor analysis is fundamental in the analysis of multiple-choice items.A good item offers equally attractive alternatives to students who do not know the answer (Husremović, 2016).Distractors should be clearly defined so that they are not confusing to higher-achieving students.The total number of distractors is 108.A distractor that was chosen by less than 5% of students was considered a non-functional distractor (NF-D) (Tarrant et al., 2009).According to this criterion, we detected 13 NF-Ds.We will only consider the distractors chosen by at least 35% of the students as indicators of students' difficulties and misconceptions (Mešić et al., 2019).We have detected six items that met this criterion (Table 5).
Distractor 'c' in item A4 was chosen by as many as 36.3% of students, while the correct alternative has been chosen by 35% of students.Based on the results, we can conclude that most students cannot recognize the graph of an odd function or its symmetry.Based on the answers in item A8, we can conclude that most students know how to determine the point of the local minimum and local maximum from the graphs, but they do not know how to determine the inflection point.By choosing the distractor 'c', most students think the inflection point is on the -axis.In item A9, most students chose the distractor 'b', while only 15% of students chose the correct answer.As item C9 is also related to the concepts of convexity and concavity, most students do not understand the connection between the sign of the second derivative of the function with the convexity/concavity of the function.Although they were required to choose a concave graph in one interval and convex in another based on the second derivative sign, most students chose a graph of the increasing function.Based on the results in item B6, we see that most students do not know how to determine the values of the function limit based on graphs.They did not recognize that the function has a horizontal asymptote y=0, so the limit of the function would be zero.Most students do not understand the algebraic notation of an increasing function (item C7), so they did not even choose a graph of an increasing function.Consequently, they have not connected the algebraic and graphic representation of the increasing function.

Students' Success on the Test and Understanding of Concepts
By analyzing students' explanations provided for the selected answer and their success on the items, we gained insight into their understanding of the functional concepts.The test was successfully solved by 32.5% of students (their total test score is higher than 18).The first question in each card referred to recognizing the type of function (square, exponential or cubic).Generally, students were successful in all four items, but they achieved the best results in items describing quadratic function and were less successful in items with exponential and cubic functions.
When it comes to the understanding of the concept of zero of the function, most students understand graphical, algebraic, and verbal representation of the zeros of the functions.Students who chose the correct answer mainly provided the correct explanations.Students who chose the wrong alternative mainly provided the following wrong explanations: 'the graph has three zeros because it intersects the -axis at two points and the y-axis at one point', 'the function must pass through the origin'.In item D2, where an algebraic expression for a rational function was given, 16 students based their explanation of the incorrect alternative on the denominator, claiming that the zeros of functions are examined using denominators and equating the denominators to zero.
In items referred to the concept of an even function, the best result was achieved in item A3 (60% of correct answers), in which they were required to select the graph of the even function.45% of students selected the correct graph for the offered algebraic formula of the even function [f(-x)=f(x)].49% of students induced correctly the parity of the function based on the algebraic notation of the function.Some students provided the following reason of their incorrect alternative: 'it passes through the origin', 'it intersects the x-axis in the origin', 'it is symmetrical with the x-axis', 'it must not be negative, and the graph does not go below x-axis', 'it does not touch the x-axis'.We can conclude that some of the students associated the concept of parity with the zero and sign of the function.Most students did not answer correctly in items that referred to the concept of an odd function.Only 18% of students realized that the graph of the odd function had the following property f(-x)=-f(x).We can also witness that the graph of the odd function is not observed from the aspect of symmetry.
In items that referred to the sign of a function the best results were achieved in item A5 (86% of correct answers), where they were expected to verbally conclude from the graph at which intervals the function was positive/negative.The worst results were obtained in item D5 (51% of correct answers), where they were expected to deduce the sign of the function by calculation based on the algebraic formula of the function.The reason for such a result may be seen in the fact that students were expected to conclude a calculation procedure.
When it comes to the understanding of the limits and asymptotes of the function, only 34% of students successfully found the asymptotes in a graph, while only 33% determined the asymptotes based on the algebraic expression of the rational function.Only 25% of the students were able to determine the values of the limit from the function's graph.Most students could not connect the limits of the function from the provided graph with marked asymptotes.Furthermore, most students (64%) do not know how to determine which graph corresponds to the calculated limit values, although asymptotes are marked on the graphs (item C6).
Most students (83%) answered correctly in the item requiring conclusion from the presented graph in which intervals of the function increases and decreases, but they still do not associate the flow of the function with the sign of the first derivative.We see this in item B7, where the same function graph is given as in item A7.Now, they were expected to conclude from the graph the sign of the first derivative.Also, most of the students (68%) do not understand the algebraic notation of function growth because, in item C7, they were expected to choose a graph of an increasing function since an algebraic notation of an increasing function with an additional representation of the first derivative was given.Only 33% of students chose the correct alternative in item D7, where the function was represented through algebraic notation, and they were assumed to conclude through a calculation process.However, most students chose the correct alternative by guessing, while others entered the point values from the offered intervals.Only two students came to a conclusion based on the first derivative value.
Most students were successful in items C8 and B8 relating to the extremes of the function.Although in item A8, the same graph was given in item B8, 43% of students knew how to describe in words the minimum, maximum, and inflection points from the graph.They were less successful since it was necessary to determine the inflection point.Also, 12 students stated the extremes points correctly but made a mistake on the inflection point (some thought that it must be on the -axis).In item D8, only two students chose the correct alternative based on the first derivative calculation, while the rest chose a correct alternative by guessing or using the point values.
In items referred to the convexity/concavity of the function students achieved the worst results.Only 16% of students concluded on a given graph at which intervals the function was convex and at which concave.The most common mistake was that they looked to the point (D) on the -axis and not to the inflection point (C) and that they substituted the terms of convexity and concavity.Only 18% of students concluded from the graph in which intervals second derivative of function is positive and negative.Only three students stated the connection between the second derivative and the convexity/concavity of a function.Only 21% of students took the correct graph based on the sign of the second derivative, but only four students determined the second derivative sign, which determined whether the function was convex/concave.Others guessed or gave incorrect explanations, most often observing whether the graph is below or above the -axis (in terms of the sign of the function).For the algebraic representation of the function, only three students concluded at which intervals the function was convex and at which concave.They explained that their conclusion was based on the second derivative value, while others guessed the answer.

Figure 5 .
Figure 5.The average difficulty level of the item according to students' opinions

Table 1 .
Items and concepts related to function

Table 2 .
Professors' suggestions for improving items

Table 3 .
Questionnaire for each item Underline every word (including the answers offered) you don't understand or you are not familiar with.Is any part of the question or the answers offered confusing?If your answer is yes, can you explain what is confusing:

Table 4 .
Item descriptive statistics for the CURF

Table 5 .
A list of the most frequently chosen distractors