Welcome to Research Studies, please consider becoming involved in some studies and learning more about Educational Technology and Computer Education.
AI Effect on Engagement
Welcome to the AI-supported Education project, where three studies are being conducted on how AI can support teaching, learning, and assessment:
AI support for teaching study, exploring how AI tools can be used to support lesson plan writing by initial teacher education students;
AI support for learning study, exploring how AI tools can be used to support cognitive processes during a postgrad course;
AI support for assessment study, exploring how AI tools can be used to complete, assess and report on assessment activities; and
AI use effects on student academic engagement.
The AI supported education research project incorporates a range of studies conducted with cohorts of students. The remaining studies draw upon other sources of data to explore AI supported Education and contribute to a wider Meta Study of Educational Technology.
Each study will begin with an exploration of the research literature followed by a study exploring the research question, followed by extension studies to explore the research question in more detail and verify the findings.
Collectively, each study is part of a project, and the projects develop a systems simulation to model the interactions between various aspects of educational technologies and computer education, culminating in a metastudy simulation of the field.
Studies in development and research will be conducted with student cohorts.
Dr Jason Zagami
AI support for teaching study has two cohorts each year, a cohort n~15 of secondary ITE students over two semesters, including an industry placement, and a cohort of n~250 primary ITE students over a 12-week period.
AI support for learning study has three cohorts each year, each of ~10 postgraduate students undertaking studies in educational technologies over a 12-week period.
AI support for assessment study has six cohorts each year, three of ~10 postgraduate students undertaking studies in educational technologies over a 12-week period, a cohort of n~15 of secondary ITE students over two semesters, and a cohort of n~250 primary ITE students over a 12-week period.
AI Effect on Engagement study
AI use effects on student academic engagement study is an open survey of university students.
Survey - Students (University - Undergraduate to Postgraduate)
Researchers and PhD students are welcome to request to join the research project team. You are asked to familiarise yourself with the Theoretical Perspective and Methodology used in the project, and you will be expected to contribute a profile of yourself to the Researcher Model (You may also wish to conduct a Self Study of your involvement in the project). You can then select which of the models you wish to contribute to and the associated studies that you will be involved in.
Coordinating and writing meetings are organised by each project team, with intensive writing periods generally occurring in June/July and December/January periods depending on the particular studies.
Request to join the project team
Project team members (Restricted Access)
Researchers agree to abide by the Australian Code for the Responsible Conduct of Research and make a significant intellectual or scholarly contribution to the research and its output and agree to be listed as an author.
Researchers to be listed as authors in research outputs will be expected to contribute to the:
analysis or interpretation of research data; and
drafting significant parts of the research output and assisting in the critical revision of team member contributions.
The order of authorship will reflect the relative contribution of the researchers who collaborated on the project and research output, determined by team members in an anonymous conjoint pair ranking process.
Researchers agree to treat fellow researchers and others involved in the research fairly and with respect, raising any concerns or conflicts of interest, and clearly informing team members of intentions in regular team meetings.
Any disputes within the project team should be resolved by the team where possible, but agree to follow the formal Griffith University institutional processes to resolve disputes if necessary, including mediation.
All Research Studies projects comply with the Australian National Statement on Ethical Conduct in Human Research. You should only provide your consent to participate in a study when fully informed of what the study involves.
Where data matching is required, e.g. from multiple surveys, only the minimum information that is needed to identify you is collected using a unique contact identifier (UCI) to minimise the risk that this information is associated with you.
In some studies, you may elect to be identified as a participant in the study on the project website and/or in publications, but this is never a requirement. Identifying information is otherwise deleted when no longer required, and all other research data is deidentified and made available in open data repositories for reuse.
This research is being led by:
Dr Jason Zagami (Griffith University) 0755528454 j.zagami@griffith.edu.au
who can answer any questions you have about the studies.
Each study involves a different research process, described in the study's invitation and informed consent, and you should only complete a request to participate in a study (or participate anonymously) once you are fully informed of what is involved in the study.
GU Ref No: 2023/44
GU Ref No. 2024/846 Engagement (Consent & Recruitment)
Each study is conducted in accordance with the Griffith University Research Integrity framework and all studies involving participants have been approved by the Griffith University Human Research Ethics Committee.
If you have any concerns or complaints about the ethical conduct of the research project, you should contact the Manager, Research Ethics (07) 37354375 or research-ethics@griffith.edu.au
Using the Self Study framework to document participation in the project and publish a study on reflective practice and researcher experiences for academic review in December, with feedback expected in March (review comments (3 months)) with publication in May (6 Months).
In February, a Systematic Literature Review Study will revise the Domain Model and evidence gaps.
A Systematic Literature Review will be submitted to a journal as a revision of the study's Systematic Literature Review for academic review in March, with feedback expected in May (review comments (3 months)) with publication in August (6 Months).
From March - October for the Secondary ITE cohort (24 weeks), the study will monitor students learning to construct lesson plans using Large Language Model (LLM) generated responses for Lower Secondary computing education from March - May, during industry placement (practicum) in June, and for Upper Secondary computer science in July - October.
From July - October for the Primary ITE cohort (12 weeks), the study will monitor students learning to construct lesson plans using Large Language Model (LLM) generated responses for primary school Technologies education.
An Expert Model and Influence Model will be developed from the study.
From March - June, for two Post-grad cohorts (12 weeks), the study will monitor student use of AI tools to complete a range of cognitive tasks.
From July - October, for a Post-grad cohort (12 weeks), the study will monitor student use of AI tools to complete a range of cognitive tasks.
An Expert Model and Influence Model will be developed from the study.
From March - October for the Secondary ITE cohort (24 weeks) and three postgrad cohorts, and from July - October for the Primary ITE cohort (12 weeks), the study will monitor student use of Large Language Model (LLM) generated responses to assessment tasks; the effectiveness of LLM feedback on student-generated responses; and the accuracy of LLM assessment of student responses against rubric responses compared to human assessments.
An Expert Model and Influence Model will be developed from the study.
The AI support for teaching, AI support for cognition, and AI support for assessment studies will be submitted to a journal for academic review in August, with feedback expected in October (review comments (3 months)) with publication in January (6 Months).
In August, researchers will conduct an Overview Study to develop a connection circle and causal relationships from the Expert and Influence Models and develop a Social Ecological Model to categorise and associate causal relationships in an Interaction Model.
The Overview Study will be submitted to a journal for academic review in October, with feedback expected in December (review comments (3 months)) with publication in March (6 Months).
In September, a Simulation Study will refine a simulation from the Interaction Model, and explore a range of simulations of the project research questions using the Process Model.
The Simulation Study will be submitted to a journal for academic review in November, with feedback expected in January (review comments (3 months)) with publication in April (6 Months).
In Early January, researchers in the Validation Study will explore the effectiveness of the Process Model simulation to replicate the results of the AI support for teaching, AI support for cognition, and AI support for assessment studies.
The Validation Study will be submitted for academic review in February, with feedback expected in April (review comments (3 months)) with publication in July (6 Months).
In February, researchers in the Meta Study will meet to combine the Process Models, informed by Validation Models, of a range of studies to refine a dynamic simulation model of the field into a Meta Study Model.
The Metastudy research team will coordinate a range of simulation explorations of the field using the Meta Study Model and allocate team members to develop studies, with drafts of these studies completed and shared by Late May.
In June, researchers will meet again to combine individual studies into an edited publication or field study, with sections allocated for completion and sharing by Late July and final edits by Late August.
The Metastudy will be submitted for academic review in September, with feedback expected in December (review comments (3 months)) with publication in March (6 Months).
The AI Effect on Engagement study is an open online survey that collects student engagement data from university students.
March - October: Secondary ITE cohort (24 weeks)
July - October: Primary ITE cohort (12 weeks)
March - June: 2 x Postgrad cohorts (12 weeks)
July - October: Postgrad cohort (12 weeks)
March - October: Secondary ITE cohort (24 weeks)
March - June: 2 x Postgrad cohorts (12 weeks)
July - October: Primary ITE cohort (12 weeks)
July - October: Postgrad cohort (12 weeks)
December - March: Tertiary students (12 weeks)
Survey - Students (Tertiary - Undergraduate to Postgraduate)
The study is a post-positivist exploration of the field of education with a focus on the application of technology to teaching and learning processes.
Accepting that the theories, hypotheses, background knowledge, and researchers' values will influence what is observed, the study pursues objectivity by recognising the effects of such biases and uses a combination of quantitative and qualitative methods to mitigate bias and improve the validity of research outcomes.
Ontologically, reality exists, but our understanding of it is imperfect and socially constructed.
Axiologically, while bias is undesirable, it is also inevitable, and mitigation is required, but this in itself will be influenced by the values and beliefs held by researchers.
(Optional) To stay informed of project calls to participate, recommendations, workshops, symposiums, presentations and publications, subscribe to the AI Supported Education project newsletter with your Unique Contact Identifier (UCI) to maintain your anonymity. No more than one email a month.
Overall Research Studies project posts will be made to the Research Projects Facebook and LinkedIn groups.
The project incorporates ten methods of research inquiry (Self Study, Systematic Literature Review, Delphi Study, Confidence Intervals, Social-Ecological Modelling, Simulation Study, Quasi-Experiemental, Most Significant Change, Validation Study, and Mixed Method) that contribute to a metastudy that draws from the ten research projects.
Focused studies (AI support for Teaching, Learning and Assessment) use Quasi-Experiemental research to contribute actors, influences and relationships to Expert and Influence models, and instead of the refinement process of the Change study used in other projects, using confidence factors to support validation.
Participants are undergraduate and postgraduate students with studies incorporated into specific courses.
The studies use a quasi-experimental pre-post research methodology to explore the research questions, with a document analysis method used to identify actors and their properties in an Expert Model and provide weightings for the influence of these properties on other actors to inform an Influence Model.
The Trend studies contribute to a systems simulation methodology identifying actors and their properties in an Expert Model and providing weightings for the influence of these properties on other actors in an Influence Model.
The Overview study categorises the relationships between actors in the Expert and Influence model into a Social Ecological Model, which is then developed into systems simulations expressed as a Process Model.
The effectiveness of the Process Model in replicating Trend study outcomes is used to validate the model in a Validation study.
The models developed in each of the ten research studies projects are finally combined into a Metamodel detailing aspects of the field of educational technologies and computer education.
Each research project is replicated annually to reflect changes in the field of educational technologies and computer education, and will involve the same methods but with an overall changed population of participants. The specific longitudinal analysis will be conducted in the Changes Study.
The AI Effect on Engagement study is a survey of university students and their engagement with AI technologies in academic study using a modified Engagement vs. Disaffection with Learning (EvsD) instrument developed by Skinner, Kindermann, and Furrer (2009). It is widely used in educational psychology to measure student engagement and its counterpart, disaffection, in learning contexts. The tool is particularly valuable for examining both positive engagement and disengagement behaviours and emotions, making it a comprehensive choice for understanding student attitudes toward learning. While gathering data on the engagement of students at all academic levels, the study focuses on the impact of AI use on the engagement of academically high-performing students as a result of AI use to support their learning.
To better understand a complex research study, self-study or reflective practice research provides a mechanism for analysis of the researchers' theories, hypotheses, background knowledge, experiences and values as the research unfolded and assists in understanding the influence these had on what is observed.
Application of reflective analysis of research practice in Self Study.
A Researcher Model is generated from the Self Study narrative as a template for modelling other researchers as Actors within the Process and Metastudy models and enables the influence of researchers to be incorporated into these models.
A systematic examination of academic literature to establish a Domain Model and Evidence Gap Map. While focusing on white (peer-reviewed research) literature, grey literature (primarily reports) is included where it is deemed significant but is not treated systematically.
A systematic literature review with meta-analysis on reported research from 2000-2022: conducted using PRISMA 2020 framework using The Lens agglomeration database.
Separate reviews are conducted on international literature and Australian-only literature and used to inform participants in the Trend Study of previous studies on trends and assist in the current study.
Systematic Literature Review on AI Supported Teaching
Overview of the research questions derived from Systematic Literature Review and generated summaries (using Generative Pre-trained Transformer) of identified topics and themes.
Details of evidence derived from peer-reviewed research and reports
Details Evidence Gap Map
Three AI Supported Education studies are being conducted on how AI can support teaching, learning, and assessment:
AI support for teaching study, exploring how AI tools can be used to support lesson plan writing by initial teacher education students;
AI support for cognition study, exploring how AI tools can be used to support cognitive processes during a postgrad course; and
AI support for assessment study, exploring how AI tools can be used to complete, assess and report on assessment activities.
March
Secondary ITE cohort (24 weeks), the study will monitor students learning to construct lesson plans using Large Language Model (LLM) generated responses for Secondary computing education
October
Primary ITE cohort (12 weeks), the study will monitor students learning to construct lesson plans using Large Language Model (LLM) generated responses for primary school Technologies education.
An Expert Model and Influence Model will be developed from the study.
March
For two Post-grad cohorts (12 weeks), the study will monitor student use of AI tools to complete a range of cognitive tasks.
July
For a third Post-grad cohort (12 weeks), the study will monitor student use of AI tools to complete a range of cognitive tasks.
An Expert Model and Influence Model will be developed from the study.
An Expert Model is generated from the AI support for teaching, AI support for cognition, and AI support for assessment studies to identify Actors (aka factors/elements/entities/objects) and Attributes (aka properties/variables) that influence the research questions. [Generated within AllOurIdeas]
From March - October for the Secondary ITE cohort (24 weeks) and three postgrad cohorts, and from July - October for the Primary ITE cohort (12 weeks), the study will monitor student use of Large Language Model (LLM) generated responses to assessment tasks; the effectiveness of LLM feedback on student-generated responses; and the accuracy of LLM assessment of student responses against rubric responses compared to human assessments.
An Expert Model and Influence Model will be developed from the study.
The amount of influence each Actor has on various Attributes is determined from a correlative Influence Model achieved from analysis of AI support for teaching, AI support for cognition, and AI support for assessment studies.
Systems research is conducted using the Expert and Influence Models to construct an Interaction Model of causal relationships in the research questions and categorise these actors and their relationships within a Social-Ecological framework.
Development of a Connection Circle of relationships between the actors identified in the Influence Model and subsequent identification of Causal Relationships (Loops) between actors.
From causal relationships, groupings of actors are derived from those actors with the strongest relationships. Generated as an onion diagram using the Social-Ecological Modelling (SEM) framework, with bands of Actors classified, grouped and clustered within bands.
A dynamic simulation model of changes in the research questions is developed from previous models to enable a range of scenarios to be explored and better understand the interaction between factors.
A systems model is developed from the Expert, Influence, and Interaction models, using the Insight Maker simulation software to create a dynamic simulation of the actors and their influence.
Using the causal relationships of actor attributes generated from connection circles and causal loop diagrams, informed by the larger relationships identified by the Social Ecological Model, a simulation model showing dynamic interactions between actor properties is generated. [Link to Agent-Based Simulation Model in Insight Maker]
Comparing the AI support for teaching, AI support for cognition, and AI support for assessment study outputs with the outputs of the Process Model simulation to validate the simulation effectiveness in modelling changes.
January
Researchers will explore the relationships identified in the Process Model [Link to Process Model simulation in Insight Maker] and the influence each actor has on the attributes of others and simulate the changes identified in the AI support for teaching, AI support for cognition, and AI support for assessment studies using the Process Model, contrasting the outcomes to validate the simulation.
The Process Model simulation is validated using a Most Significant Change (MSC) research framework on changes identified in AI support for teaching, AI support for cognition, and AI support for assessment studies with simulation comparisons used to compare identified changes with those generated by the simulation model. Through the hierarchical Most Significant Change process, change cases are grouped and categorised to determine those which are the most significant to research questions.
Models produced from various studies are integrated into a larger Meta Model researching the interactions between various aspects of Education Technologies.
The Simulation Model produced by the Metastudy enables a contextualisation of the studies and a validation process on the predictions from the study over time with the predictions generated from the Metastudy simulation model.
A Metastudy model is developed to simulate field-level interactions of a wide range of actors and attributes. [Link to Agent-Based Simulation Metastudy Model in Insight Maker]
TBA late 2023
TBA late 2023
Annual online symposia in early December to present research findings.
Online Writing Workshops in June/July and December/January to support research team publications.
TBA 2023
TBA 2024
International AIED Society €25
AIED Conference (July)
International Journal of Artificial Intelligence in Education (Springer $2790)
TBA 2024