Adjustment of the examination format

Problem:

Students can achieve positive examination results with the help of generative AI, even without their own work, if examination formats are used without prevention or if there are no technical or organizational measures in place. 

If the use of generative AI tools is prohibited by teachers, it is not possible to check whether students are complying with this without technical or organizational measures. Reliable technical identification of the use of AI tools is hardly possible.
 

Approaches - Depending on the didactic concept of the course, the appropriate approach must be selected:

  • On-site examinations - oral, handwritten, secure digital examination mode on site (within the TU Wien premises)
  • Combined examinations - additional elements can be added to (un)supervised written examinations (e.g. plausibility check of examination answers through oral questions, short oral examinations or written reflections)

  • Exam formats are redesigned so that they cannot be solved exclusively by AI outputs:

    •  Exams with learning outcomes at a higher level: generative AI tools can be used primarily for low learning outcome levels (remember, understand). Here, the answers are often good to very good. The higher the learning outcome level, the more difficult it is to use AI tools to generate solutions directly. It is therefore advisable to test questions at a higher level (e.g. do not test learning content in the form of factual knowledge, but aim to apply what has been learned in practical and complex situations).

    •  Assessment at all levels - in addition to an examination performance, which can also represent a learning product, for example (e.g. as implementation in the context of a project), the learning process is also assessed; this can be accompanied by didactic methods such as project-based learning, case-based learning, problem-based learning. Example of a possible implementation: Several tasks are set over the course of a semester. In addition to texts, these can also include multimedia content such as visualizations, collages, etc. The learning products can be discussed together in class.

Feedbackblasen

© Pixabay | Shari Jo

Feedbackblasen

Empfehlung: 

Ausgehend vom didaktischen Konzept und den Lernergebnissen sollte entschieden werden, welche Anpassungen beim Prüfungsformat geeignet sind. Je nach didaktischem Szenario könnte sich der Aufwand erhöhen (z.B. manuelle Korrektur komplexerer Fragen, steigende Anzahl mündlicher Prüfungen, höherer Betreuungsaufwand)

Dealing with generative AI tools as a permitted resource

Problem:

 Students use generative AI tools as permitted resource and ...

  •  ... rely too heavily on the tools and hardly question the results. Any bias or false information is not recognized.
  •  ... have very different and sometimes insufficient basic knowledge of the learning content to be able to recognize any errors.
  •  ... achieve qualitatively different results due to unequal accessibility (paid versus free AI tools, accessible versus non-accessible AI tools).
     

Possible solutions:

Both teachers and students should use generative AI tools responsibly in the course. Teachers should critically question their use as part of the didactic concept and introduce students to reflective use in the course:

  • Create learning opportunities for practicing reflective and critical use (e.g. use in exercise formats and discussion in class)
  • Provide specific tasks and examination questions aimed at the critical and reflective use of AI tools (e.g. evaluation of exemplary outputs of generative AI tools):
    • Example 1: Ask the students to evaluate the quality of the output of the generative AI, identify errors or improve the output.
    • Example 2: Ask students to identify specific errors or limitations in the use of generative AI tools in relation to the subject area.
  • Provide clear guidelines for the use of AI technologies in the course: Define in advance in the course how, when and for what purpose the use of AI tools is permitted. Alternatively, you can also develop rules for use together with your students. Prof. Dr. Christian Spannagel's Rules for Tools provide an example of usage rules. To increase the binding nature of the rules, it is advisable to obtain confirmation from the students.   

  • Ensuring accessibility and fairness: Please note that the use of some AI tools, such as ChatGPT, requires registration and is sometimes subject to a fee. It can therefore not be assumed that students will use these tools. Therefore, also consider using alternative offers that are neither subject to a fee nor require registration. Information on recommended generative AI tools in the course from TU Wien will follow.

  • Usage instructions for examinations: Please indicate in advance for your exams to what extent the use of generative AI tools is permitted (see Rules for Tools). If necessary, combine this with a declaration of independence from TU Wien. (You can find a sample template in the Leitfaden zum Umgang mit Plagiaten in studentischen Arbeiten an der TU Wien, Kapitel 3.3 Einsatz von Plagiatssoftware).

  • Addressing the legal rules on data protection: In your course, raise awareness of the need to be careful when entering data into the model and to comply with the legal rules on data protection.
    Caution: Never enter personal data about yourself or other people (names, health data, e-mail address, etc.) or data that makes people identifiable. As a rule of thumb, do not use any information in prompts that you would not otherwise publish (e.g. on social media). Information on data protection regulations when using AI can be found in the Empfehlung zur guten Praxis im Umgang mit generativer KI an der Technischen Universität Wien. 

  •  Raising awareness with regard to ethics: Make your students aware that generative AI tools have a bias and address ethical issues such as equal treatment/anti-discrimination, academic integrity (trust, honesty and respect), energy consumption, false statements, unresolved copy-right issues, misuse & manipulation, etc. The AI v Ethics link collection by Prof. Dr. Peter Purgathofer provides more in-depth information.
  • Reflect on how to deal with bias and misinformation in AI technologies: Test this with your students and then reflect together on how to deal with it.
Feedbackblasen

© Pixabay | Shari Jo

Feedbackblasen

Recommendation:

It should be considered whether the use of generative AI tools makes sense from a didactic perspective: What can generative AI tools be used for and what not? (For an overview of the taxonomies of learning objectives, in which the tasks of the cognitive requirement levels that can be performed by AI are listed, see the presentation Bloom's Taxonomy Revisited, opens an external URL in a new window by Oregon State University). Specific human skills and qualifications should be emphasized and reviewed. In order to enable students to use generative AI tools in a critically reflective way, opportunities for reflection should be created in addition to learning opportunities. Rules for the use of AI technologies should be addressed in the course and can be developed and agreed upon together with students. Instructions on how to use AI technologies can be provided in advance in TISS and TUWEL.
 

Conclusion: Audit design

  • Based on the learning outcomes to be checked, decide which tools are permitted and to what extent:

    • Option 1: Prohibition of AI tools (oral or handwritten examinations; secure digital examination mode on site (within TU Wien premises)

    • Option 2: Permission or restricted permission of generative AI tools with mandatory labeling of AI-generated text passages
      It is recommended to handle the labeling like citations (name of the tool, version, date, etc.). You can find information on labeling information from TU Wien in the recommendation on good practice in dealing with generative AI at the Vienna University of Technology, What to look out for - dealing with AI in written work. A sample template for declarations of originality is included in the Guidelines for dealing with plagiarism in student work at TU Wien, Chapter 3.3 Use of plagiarism software), whereby declarations of originality can also be obtained via TUWEL.
  • Transparent communication at the beginning of the course (e.g. in the course description in TISS):
    • Students need to know in the course whether and for what they are allowed to use generative AI tools.
    • Students must receive transparent information before the examination about the framework conditions that apply to the examination. Ideally, there should be an explanation that refers once again to the applicable examination guidelines.
  • Consideration in the didactic concept of the course:
    • Due to the change in the examination format (e.g. combined examinations), this may result in additional time for the lecturers. The additional time required for examinations must be taken into account when planning the examination format and the didactic method of the course.
    • The use of generative AI tools can shift the student workload. This should be critically questioned and the examination format and the focus of the learning outcomes adapted accordingly. 

Sources of information cited in the text

Additional, further information