Where free chatbots fail in school physics: an evaluation across 780 Uzbek-language textbook problems

Abstract Pupils in Uzbekistan now consult free generative chatbots as a routine part of their physics homework, but the reliability of these systems on Uzbek-language K-12 problems has not been measured. We tested the free chat interfaces of ChatGPT (the GPT-5 family) and DeepSeek (the V3 family) on 780 worked exercises from the four current Uzbek state physics textbooks for grades 7–10. Each problem was submitted in its original Latin-script form, with textbook figures uploaded when referenced. Testing was carried out grade by grade between September 2025 and March 2026; the two systems were administered and graded by different researchers, with a 20% subsample ( n = 156) cross-checked for rubric consistency. Of the 772 problems scoreable for both models, ChatGPT solved 575 correctly (74.5%) and DeepSeek 600 (77.7%). The difference between the two systems was small and not sufficiently robust to support a pedagogical preference for one platform. More important for classroom practice, both systems failed on the same 119 problems (15.4%), and these joint failures were concentrated in identifiable textbook sections: fuel calorimetry and reference-value retrieval in grade 7, molecular-kinetic theory in grade 9, and inclined-plane dynamics with friction and temperature-dependent resistance in grade 10. Grade 8 showed a much lower joint-failure rate (4.1%). Across the main failure clusters, recurring demands included textbook-specific value retrieval, unit tracking and multi-step quantitative reasoning. The teaching implication is therefore not to choose a ‘better’ chatbot, but to teach pupils where verification is most necessary; we provide a section-by-section recommendation table for this purpose.

Hali tarjima qilinmagan