Document Type
Poster
College
Henry M. Rowan College of Engineering
Start Date
25-3-2026 1:00 PM
End Date
25-3-2026 2:00 PM
Abstract
We investigated how Generative AI (GenAI) can be used to identify core engineering competencies in entry-level job postings and how these results compare to traditional qualitative analysis approaches. Job postings represent a rich potential data source for understanding workforce expectations across engineering disciplines. Simultaneously, recent advances in GenAI show promise for scalable analysis of large text-based datasets. The purpose of this study is to identify core competencies emphasized in entry-level chemical (ChE) and mechanical (ME) engineering job postings and to evaluate the use of GenAI for this task. We analyzed 50 entry-level job postings, including 25 ChE and 25 ME positions, sourced from LinkedIn Jobs. Competencies were coded using a deductive framework based on the 16 core engineering competencies identified in Passow and Passow’s 2017 systematic review. Two parallel analyses were conducted - one by human researchers and one by GenAI. Qualitative coding of the job postings was conducted by the researchers using MAXQDA. Findings demonstrated that for both ChE and ME job postings, the following competencies were the most prevalent: apply knowledge, apply skills, communicate effectively, and coordinate efforts. Researchers then developed a structured prompt for ChatGPT aimed at coding job postings and minimizing hallucinations. Job postings previously coded by researchers were input and coded by ChatGPT based on the competencies descriptions. This project will present the initial results related to the agreement between the GenAI and researcher coding outputs using inter-rater reliability metrics, which will inform suggestions for how to refine the GenAI prompting and analysis strategy. Initial findings demonstrate promising similarities between the coding of human researchers and GenAI. This work contributes methodological insight into the use of GenAI for engineering education research and provides a scalable approach for analyzing workforce expectations to inform discipline-specific curricular design, professional development programming, and career preparation efforts that better align student training with employer needs.
Using Large Language Models to Identify Core Competencies in Entry-Level Chemical and Mechanical Engineering Job
We investigated how Generative AI (GenAI) can be used to identify core engineering competencies in entry-level job postings and how these results compare to traditional qualitative analysis approaches. Job postings represent a rich potential data source for understanding workforce expectations across engineering disciplines. Simultaneously, recent advances in GenAI show promise for scalable analysis of large text-based datasets. The purpose of this study is to identify core competencies emphasized in entry-level chemical (ChE) and mechanical (ME) engineering job postings and to evaluate the use of GenAI for this task. We analyzed 50 entry-level job postings, including 25 ChE and 25 ME positions, sourced from LinkedIn Jobs. Competencies were coded using a deductive framework based on the 16 core engineering competencies identified in Passow and Passow’s 2017 systematic review. Two parallel analyses were conducted - one by human researchers and one by GenAI. Qualitative coding of the job postings was conducted by the researchers using MAXQDA. Findings demonstrated that for both ChE and ME job postings, the following competencies were the most prevalent: apply knowledge, apply skills, communicate effectively, and coordinate efforts. Researchers then developed a structured prompt for ChatGPT aimed at coding job postings and minimizing hallucinations. Job postings previously coded by researchers were input and coded by ChatGPT based on the competencies descriptions. This project will present the initial results related to the agreement between the GenAI and researcher coding outputs using inter-rater reliability metrics, which will inform suggestions for how to refine the GenAI prompting and analysis strategy. Initial findings demonstrate promising similarities between the coding of human researchers and GenAI. This work contributes methodological insight into the use of GenAI for engineering education research and provides a scalable approach for analyzing workforce expectations to inform discipline-specific curricular design, professional development programming, and career preparation efforts that better align student training with employer needs.