LLM Code Error
AboutFeedback
Distribution of semantic characteristics of code generation errors
0%2%4%6%8%10%12%14%16%18%CodeGen-16BMissing multiple stepsIncorrect conditionMeaningless code snippetWrong (logical) directionWrong method/variableConstant value errorUndefined nameMissing one stepIncorrect arithmetic operationMissing conditionOnly commentsInfinite LoopIncorrect comparison operationPercentage
0%5%10%15%20%25%InCoder-1BWrong (logical) directionMissing multiple stepsIncorrect conditionWrong method/variableMeaningless code snippetOnly commentsUndefined nameMissing one stepIncorrect arithmetic operationConstant value errorMissing conditionInfinite LoopIncorrect comparison operationPercentage
0%5%10%15%20%25%GPT-3.5Wrong (logical) directionIncorrect conditionIncorrect arithmetic operationConstant value errorWrong method/variableMissing multiple stepsMissing conditionOnly commentsMissing one stepUndefined namePercentage
0%5%10%15%20%25%30%35%GPT-4Wrong (logical) directionIncorrect conditionConstant value errorMissing one stepMissing multiple stepsOnly commentsMissing conditionIncorrect arithmetic operationUndefined namePercentage
0%5%10%15%20%25%30%SantaCoderWrong (logical) directionMissing multiple stepsIncorrect conditionWrong method/variableMeaningless code snippetIncorrect arithmetic operationConstant value errorUndefined nameMissing one stepMissing conditionInfinite LoopIncorrect comparison operationPercentage
0%5%10%15%20%25%StarCoderMeaningless code snippetMissing multiple stepsWrong (logical) directionIncorrect conditionWrong method/variableUndefined nameIncorrect arithmetic operationConstant value errorMissing one stepMissing conditionPercentage
Distribution of syntactic characteristics of code generation errors
0%5%10%15%20%25%30%35%40%CodeGen-16BMissing code blockIncorrect code blockIf errorIncorrect function nameIncorrect function argumentsIncorrect constantFor errorIncorrect arithmeticIncorrect return valueWhile errorIncorrect variable nameIncorrect comparisonImport errorPercentage
0%5%10%15%20%25%30%35%InCoder-1BIncorrect code blockMissing code blockIncorrect function nameIf errorIncorrect function argumentsFor errorIncorrect arithmeticIncorrect return valueImport errorIncorrect constantWhile errorIncorrect variable nameIncorrect method call targetPercentage
0%5%10%15%20%25%GPT-3.5Incorrect code blockIf errorMissing code blockIncorrect function argumentsFor errorIncorrect constantIncorrect return valueIncorrect arithmeticIncorrect function nameImport errorPercentage
0%5%10%15%20%25%30%35%GPT-4Incorrect code blockMissing code blockIf errorIncorrect return valueIncorrect arithmeticImport errorFor errorPercentage
0%5%10%15%20%25%30%35%SantaCoderIncorrect code blockMissing code blockIf errorIncorrect return valueIncorrect function argumentsIncorrect function nameFor errorImport errorIncorrect constantIncorrect arithmeticIncorrect method call targetIncorrect variable nameWhile errorPercentage
0%10%20%30%40%50%StarCoderMissing code blockIncorrect code blockIf errorIncorrect function argumentsIncorrect function nameFor errorIncorrect return valueImport errorIncorrect constantIncorrect method call targetIncorrect variable namePercentage

© University of Alberta, University of Illinois Urbana-Champaign, The University of Tokyo, Purdue University