Multi-AI comparison for developing evaluative judgment in chemical engineering design: An empirical study using a deliberately small-scale ammonia plant task

Generative artificial intelligence (GenAI) tools can generate fluent yet flawed solutions to open-ended engineering design problems, challenging conventional assessment that rewards answer production. This study explores a multi-AI comparison assessment that foregrounds evaluative judgment: students’ ability to critically assess the quality, plausibility, and credibility of competing GenAI-generated designs using core chemical engineering principles. This paper reports one of the first empirical studies of such a multi-AI comparison activity in a senior chemical engineering design course. Twenty final-year students evaluated GenAI-generated conceptual designs for an intentionally small-scale ammonia production plant (1000 tonnes per year). This capacity was deliberately chosen far below typical commercial-scale ammonia plants in order to probe whether students would critically interrogate scale-dependent assumptions embedded in GenAI-generated responses. Guided by evaluative judgment theory and the ICAP framework, the study operationalizes evaluative judgment through a structured coding framework based on observable engineering criteria (mass/energy balances, reactor sizing via GHSV, recycle/purge logic, and system-level energy accounting). Analysis of student reports showed that 90% of students identified inconsistencies in reactor sizing and 55% identified mass-balance issues, whereas only 25% identified system-level energy omissions. These concrete detection rates and blind spots provide practical guidance for rubric development in GenAI-rich design courses. The findings demonstrate that multi-AI comparison elicits substantive evaluative reasoning by making uncertainty and flawed assumptions visible. This transferable assessment design shifts emphasis from answer generation to evidence-based critique and justification, better aligning chemical engineering education with professional judgment needs in the GenAI era.

Hali tarjima qilinmagan