每一處引用，
回到原文。

We introduce WebArena, a realistic and reproducible web environment designed to facilitate the development of autonomous agents capable of executing tasks (§2). An overview of WebArena is in Figure 1. Our environment comprises four fully operational, self-hosted web applications, each representing a distinct domain prevalent on the internet: online shopping, discussion forums, collaborative development, and business content management. Furthermore, WebArena incorporates several utility tools, such as map, calculator, and scratchpad, to best support possible human-like task executions. Lastly, WebArena is complemented by an extensive collection of documentation and knowledge bases that vary from general resources like English Wikipedia to more domain-specific references, such as manuals for using the integrated development tool (Fan et al., 2022). The content populating these websites is extracted from their real-world counterparts, preserving the authenticity of the content served on each platform. We deliver the hosting services using Docker containers with gym-APIs (Brockman et al., 2016), ensuring both the usability and the reproducibility of WebArena.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

challenges such as bots being subject to CAPTCHAs, unpredictable content modifications, and configuration changes, which obstruct a fair comparison across different systems over time. We achieve realism by using open-source libraries that underlie many in-use sites from several popular categories and importing data to our environment from their real-world counterparts.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Specifically, given a natural language test requirement, WebTestPilot decomposes it into n (condition, action, expectation) steps. For each step, WebTestPilot translates the condition and expectation into pre- and post-condition assertions. It then applies symbolization to extract relevant UI components as symbols, which are composed via a DSL to construct executable assertions satisfying the specified constraints. To support cross-state reasoning, WebTestPilot uses page reidentification to detect revisited pages and maintain a structured history of test states.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

5.1.1 Web Applications. We search GitHub for open-source web applications and select those based on five criteria: (1) popularity, with ≥5,000 stars; (2) active development, with >50 contributors and >1,000 commits, and a commit in the past month; (3) maturity, publicly available for >5 years; (4) practical relevance, indicated by active deployment, recognizable domain or organization, commercial support, or adoption by well-known entities; and (5) user-facing documentation describing core features. We select the following four web applications: • BookStack [8]: A hierarchical documentation management platform with rich text editing. • Indico [24]: An event manager for conferences, meetings, and lectures. • InvoiceNinja [25]: A business-oriented invoicing platform with multi-step workflows. • PrestaShop [50]: A full-stack e-commerce platform with store management feature. We package the applications into reproducible Docker Compose environments.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

5.1.4 Injected Bugs. We design a single artificial bug bugij: S →S for each test requirement Dij. These bugs induce incorrect behaviors while ensuring stable and reproducible experiments by locking application versions. To ensure realism, we examine closed GitHub issues labeled "Bug" from each application repository. From a total of 2,043 issues, we randomly sample 10%. We perform open coding on the titles and descriptions of the sampled issues to identify meaningful labels, and then conduct a thematic analysis to group these labels into broader bug categories. Two co-authors independently perform the analysis, with a third resolving any disagreements. We exclude crash bugs and purely cosmetic bugs (e.g., layout or positioning issues) that do not affect functionality, as prior work has already addressed them. Based on our analysis, we focus on four categories: • Missing UI elements: Required interface components are absent, breaking feature functionality. For example, in prestashop/#22170, the "Configure" button is missing for newly installed modules. • Data inconsistency: Information shown to the user does not match expected values. For example, in indico/#5197, the category search results include items that were previously deleted. • No-op actions: User actions fail silently or have no effect. For example, in invoiceninja/#11188, the filter button in "Customer > Documents" does not sort or filter and always shows the full list. • Navigation failures: Pages fail to transition correctly. For example, in prestashop/#14796, a logged-in user selecting any option in the back-office menu is redirected to the login page.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Interestingly, our results show that LLMs are more likely to generate test oracles that capture the actual program behaviour (what is actually implemented) rather than the expected one, i.e., the intended behaviour. Additionally, we find that the overall performance of the LLMs is relatively low (less than 50% accuracy) meaning that LLMs do not provide a strong oracle correctness signal. Therefore, all LLMs suggestions will need human inspection. Taken together, our results corroborate the conclusion that unless having meaningful test or variable names LLMs can mainly be used to capture the actual program behaviour (thus to be used for regression testing). Additionally, we find that LLMs could be a good addition to existing test generation tools, or to the test writing task, by using them to perform test augmentation. Overall, this work raises the awareness of the practical issues involved, advantages and disadvantages of the LLM-based test oracle generation abilities.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Abstract—Automated unit test generation aims to improve software quality while reducing the time and effort required for creating tests manually. However, existing techniques primarily generate regression oracles that predicate on the implemented behavior of the class under test. They do not address the oracle problem: the challenge of distinguishing correct from incorrect program behavior.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

1Oracles generated with more context compile and detect bugs more reliably. CUT-level context significantly outperforms other configurations, achieving 53.64% accuracy versus 40.74% (MUT) and 40.38% (test prefix only). This is an expected result. 2Prompting style matters: zero-shot and few-shot prompts yield higher compilation rates (67.38% and 72.96%) and accuracy (54.56% and 51.30%) than CoT and ToT, which struggle with low compilation (both below 50%). 3Incorporating the CUT in the input prompt, along with zero-shot and few-shot prompting techniques, leads to the most consistently accurate LLM-generated test oracles. However, our findings show there is potential for reasoning based prompt techniques like CoT and ToT to be able to produce accurate test oracles given their high accuracy when they do produce compilable assertions.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

5Prompting strategy has a stronger impact on oracle effectiveness than LLM choice. These findings suggest that prompt design and context play a critical role in the effectiveness of LLM-based oracle generation. While reasoning-driven prompting (e.g., CoT, ToT) shows potential when it compiles, zero-shot and few-shot prompting currently offer the best tradeoff between accuracy and robustness. Our study offers guidance for AI-assisted testing tools usable by both testing and prompt experts in the FM era.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Abstract—Large language models (LLMs) increasingly decide whether software behaves correctly, either by writing a test oracle or by acting as one. Yet two oracles can look identical and rest on different ground: one assertion encodes a written specification, another only what the model learned in training. Prior secondary studies sort oracles by form or by technique, rarely by the property that governs how far a verdict can be trusted: where its authority comes from. This systematic literature review, reported under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines, screens 2,436 records to 54 included studies, extended by citation searching (snowballing) to 83 in total. We read the corpus along three axes: the source of an oracle’s authority, the form it takes, and the mechanism that adjudicates it. Just over half of the corpus reaches a verdict with no specification at all. That is what lets these oracles work on code with no specification to consult, and what leaves a challenged verdict with less to fall back on. Source and mechanism cross-cut rather than coincide, so a label such as LLM-as-a-judge names how a verdict is produced, not why it should be trusted. Oracle quality is most often judged by resemblance to a known oracle rather than by whether injected faults are caught. The first question to ask of any LLM oracle is therefore what one would point to in defending its verdict. The protocol, search query, and per-study coding sheet are released.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

We identify two fundamental challenges that prevent existing GUI agents from effective exploratory testing: (i) Goal-Oriented Masking. Most GUI agents are optimized to maximize task success rates, which inherently encourages robustness against environmental obstacles. In a testing context, this goal-oriented nature leads the agent to perceive functional anomalies as traversable hurdles rather than reportable defects. As shown in Figure 1(a), when encountering a non-responsive button, the agent’s policy autonomously seeks alternative navigation paths to reach the goal. This “success-at-all-costs” behavior effectively masks the defect, rendering it invisible to the quality assurance pipeline. (ii) Execution-Bias Attribution. Exploratory testing lacks explicit oracles, requiring agents to distinguish between their own operational failures (e.g., coordinate miscalculations) and genuine software defects. Due to the stochastic nature of MLLM interactions, current agents exhibit a systematic bias toward self-attribution: erroneously assuming that any failure to trigger a state change stems from their own execution imprecision. As illustrated in Figure 1(b), GUI-Owl misinterprets a system-level rendering failure as a misaligned click, causing the genuine defect to be misclassified as a transient execution error in the logs.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Exploratory GUI testing is essential for software quality but suffers from high manual costs. While Multi-modal Large Language Model (MLLM) agents excel in navigation, they fail to autonomously discover defects due to two core challenges: Goal-Oriented Masking, where agents prioritize task completion over reporting anomalies, and Execution-Bias Attribution, where system defects are misidentified as agent errors. To address these, we first introduce GUITestBench, the first interactive benchmark for this task, featuring 143 tasks across 26 defects. We then propose GUITester, a multi-agent framework that decouples navigation from verification via two modules: (i) a Planning-Execution Module (PEM) that proactively probes for defects via embedded testing intents, and (ii) a Hierarchical Reflection Module (HRM) that resolves attribution ambiguity through interaction history analysis. GUITester achieves an F1-score of 48.90% (Pass@3) on GUITestBench, outperforming state-of-the-art baselines (33.35%). Our work demonstrates the feasibility of autonomous exploratory testing and provides a robust foundation for future GUI quality assurance 1.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

5 diverse domains. Using the exploratory task synthesis strategies described above, we expand these scenarios into 143 navigation tasks. The detailed distribution across defect types and application domains is shown in Figure 3. Based on defect-triggering mechanisms, defects fall into two categories: single-action defects, which are triggered by one action on a specific state (62.24%), and multi-action defects, which require a sequence of prerequisite actions (37.76%).

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Abstract—This paper presents Tuco, a prototype framework that uses large language models (LLMs) to support web acceptance testing. In current software development, requirement-level acceptance still relies heavily on manual effort, while end-toend (E2E) automated tests demand executable scripts whose assertions must reflect real requirements. Tuco constructs a staged artifact pipeline that produces acceptance rules, web operation manuals, Gherkin scenarios, Playwright E2E tests, and operation recordings. Its browser execution stage adopts an orchestrator-executor design: the orchestrator dispatches scenarios and aggregates artifacts, while executor agents operate the browser, evaluate observed behavior, and generate test scripts or candidate-defect evidence. A central design principle is Gherkin-guided assertion strictness: generated Playwright assertions should verify the observable outcome stated in the Gherkin Then clause, and weakened assertions are reported as audit findings rather than silently accepted as passing tests. In a post-fix controlled rerun over 63 scenarios across four modules, together with an audit ablation over 12 pre-specified scenarios, Tuco surfaced candidate defects related to data integrity, permission protection, error handling, and specification mismatch. To verify whether candidate items are real product defects and reduce model-verdict bias, three reviewers with about five years of practical experience retrospectively triaged the candidates under a majority-vote rule; we report both Tuco-raw and reviewermajority confirmed results.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Tuco organizes acceptance work around four artifacts: extracted rules, a web operation manual, Gherkin scenarios, and browser-executed evidence. When the UI path, expected behavior, or assertion target is underspecified, the browser execution records the gap in an execution audit report that serves as input for orchestrator or human review. A. Rule Extraction Rule extraction normalizes requirement documents into acceptance rules that can be cited during browser execution. These documents may include feature descriptions, field constraints, role permissions, error-handling rules, data-state transitions, and exceptional cases. Tuco uses an LLM to organize these natural-language requirements while retaining the source context for each rule.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

C. Gherkin Acceptance Scenario Generation Gherkin scenarios are drafted from the rules and operation manual. The Given, When, and Then structure separates state, action, and expected behavior. Tuco uses Gherkin as the source for later Playwright assertions. Given describes the precondition, When describes the user operation, and Then describes what the system must satisfy. The Then clause becomes the basis for the generated Playwright assertion. For example, if a Then clause says that the system must reject a duplicate email address and display a clear error message, a later Playwright test cannot merely check that an API returned an error or that the page is still visible. It must verify that the user-visible feedback appears and that the feedback corresponds to the duplicate-email condition.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

We contribute with (1) a benchmark of three offline web applications, and a suite of 113 manual test cases, split between passing and failing cases, to evaluate and compare ATAs performance, (2) SeeAct-ATA and pinATA, two open-source ATA implementations capable of executing test steps, verifying assertions and giving verdicts, and (3) comparative experiments using our benchmark that quantifies our ATAs effectiveness. Finally we also proceed to a qualitative evaluation to identify the limitations of PinATA, our best performing implementation.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

The AER rate describes the number of time the agent failed before the human without even having seen the failing step. These errors are likely due to the agent’s limitation in exploring the website or validating an assertion. Hallucinations are produced outputs that are coherent and grammatically correct but factually incorrect or nonsensical. In that case, hallucinations would be unlawfully validated step assertions and the HER quantify them.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

End-to-end web testing is challenging due to the need to explore diverse web application functionalities. Current state-of-the-art methods, such as WebCanvas, are not designed for broad functionality exploration; they rely on specific, detailed task descriptions, limiting their adaptability in dynamic web environments. We introduce NaviQAte, which frames web application exploration as a question-and-answer task, generating action sequences for functionalities without requiring detailed parameters. Our three-phase approach utilizes advanced large language models like GPT-4o for complex decision-making and cost-effective models, such as GPT-4o mini, for simpler tasks. NaviQAte focuses on functionality-guided web application navigation, integrating multi-modal inputs such as text and images to enhance contextual understanding. Evaluations on the Mind2Web-Live and Mind2Web-Live-Abstracted datasets show that NaviQAte achieves a 44.23% success rate in user task navigation and a 38.46% success rate in functionality navigation, representing a 15% and 33% improvement over WebCanvas. These results underscore the effectiveness of our approach in advancing automated web application testing.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

This article describes a novel technique for crawling AJAX-based applications through automatic dynamic analysis of user-interface-state changes in Web browsers.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

To address these challenges, in this paper, we propose Temac, an approach that enhances automated web GUI testing using LLM-based multi-agent collaboration, aiming to maintain both exploration breadth and depth to increase code coverage. Temac is motivated by our insight that LLMs can enhance automated web GUI testing in executing complex functionalities, while the information discovered during automated web GUI testing can, in turn, be provided as the domain knowledge to improve the success rate of LLM-based task planning and execution. Specifically, given a web application, Temac initially runs an existing approach of automated web GUI testing to broadly explore application states. When the testing coverage stagnates, Temac then employs LLM-based agents to summarize the collected multi-modal information into a structured and concise knowledge base and to infer not-covered functionalities. Guided by this knowledge base, Temac finally uses specialized LLM-based agents to target and execute these not-covered functionalities, reaching deeper states beyond those explored by a testing approach without using LLMs. Our evaluation results show that Temac improves state-of-theart approaches of automated web GUI testing from 12.5% to 60.3% on average code coverage on six complex open-source web applications, while revealing 445 unique faults in the top 20 real-world web applications. These results strongly demonstrate the effectiveness and the general applicability of Temac.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

• Crawljax [3]. A model-based AWGT approach that is widely used by existing work [5]–[7], [45] as a baseline. • FragGen [6]. A model-based AWGT approach that is developed based on Crawljax, equipped with an effective state abstraction approach using screenshot matching.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

The Graphical User Interface (GUI) is how users interact with mobile apps. To ensure it functions properly, testing engineers have to make sure it functions as intended, based on test requirements that are typically written in natural language. While widely adopted manual testing and script-based methods are effective, they demand substantial effort due to the vast number of GUI pages and rapid iterations in modern mobile apps. This paper introduces AUITestAgent, the first automatic, natural language-driven GUI testing tool for mobile apps, capable of fully automating the entire process of GUI interaction and function verification. Since test requirements typically contain interaction commands and verification oracles. AUITestAgent can extract GUI interactions from test requirements via dynamically organized agents. Then, AUITestAgent employs a multi-dimensional data extraction strategy to retrieve data relevant to the test requirements from the interaction trace and perform verification. Experiments on customized benchmarks 1 demonstrate that AUITestAgent outperforms existing tools in the quality of generated GUI interactions and achieved the accuracy of verifications of 94%. Moreover, field deployment in Meituan has shown AUITestAgent’s practical usability, with it detecting 4 new functional bugs during 10 regression tests in two months.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Graphical user interfaces (GUIs) are foundational to interactive systems and play a pivotal role in early requirements elicitation through prototyping. Ensuring that GUI implementations fulfill natural language (NL) requirements is essential for robust software engineering, especially as LLM-driven programming agents become increasingly integrated into development workflows. Existing GUI testing approaches, whether traditional or LLM-driven, often fall short in handling the complexity of modern interfaces, and typically lack actionable feedback and effective integration with automated development agents. In this paper, we introduce GUISpector, a novel framework that leverages a multi-modal (M)LLM-based agent for the automated verification of NL requirements in GUI prototypes. First, GUISpector adapts a MLLM agent to interpret and operationalize NL requirements, enabling to autonomously plan and execute verification trajectories across GUI applications. Second, GUISpector systematically extracts detailed NL feedback from the agent’s verification process, providing developers with actionable insights that can be used to iteratively refine the GUI artifact or directly inform LLM-based code generation in a closed feedback loop. Third, we present an integrated tool that unifies these capabilities, offering practitioners an accessible interface for supervising verification runs, inspecting agent rationales and managing the end-to-end requirements verification process. We evaluated GUISpector on a comprehensive set of 150 requirements based on 900 acceptance criteria annotations across diverse GUI applications, demonstrating effective detection of requirement satisfaction and violations and highlighting its potential for seamless integration of actionable feedback into automated LLM-driven development workflows. The video presentation of GUISpector is available at: https://youtu.be/JByYF6BNQeE, showcasing its main capabilities.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Model #Turns #Tokens Functionality Constraint Interaction Content Overall Cov. F1 Cov. F1 Cov. F1 Cov. F1 Cov. P R F1 Open-Source LLMs Minimax-M2.1 41.7 3.58M 77.9 12.3 40.4 15.8 42.2 19.9 47.1 7.7 60.1 22.3 14.6 15.2 Qwen3-Coder-Next 63.4 6.24M 77.6 14.1 48.3 23.8 42.7 11.4 35.9 4.3 60.4 27.8 15.8 17.3 GLM-4.7 41.6 3.47M 79.9 16.5 47.3 20.5 36.0 17.2 48.9 4.3 61.1 26.7 16.6 18.1 GLM-5 41.3 3.71M 79.7 11.9 50.1 26.9 41.4 20.9 50.6 3.4 63.1 30.4 15.6 19.0 Step-3.5-Flash 57.0 3.37M 79.8 20.1 53.6 27.9 48.5 21.2 60.6 2.6 66.0 34.6 20.8 23.4 MiMo-V2-Flash 59.8 7.26M 80.0 21.9 48.7 29.2 48.0 20.3 50.3 7.3 63.5 34.8 24.6 25.1 Closed-Source LLMs Claude Opus 4.5 42.9 2.60M 83.3 18.8 50.3 21.2 40.8 14.7 42.1 6.8 63.2 33.0 16.5 20.2 Claude Sonnet 4.5 37.6 1.90M 81.0 22.2 47.7 22.5 46.7 19.9 51.6 1.7 63.7 32.1 19.7 21.9 GPT-5.2 69.5 7.43M 76.9 25.3 51.9 21.5 43.0 23.2 46.1 6.2 61.0 24.7 25.2 22.9 GPT-5.1 30.3 0.87M 76.4 30.9 51.2 26.9 49.7 22.0 57.5 15.3 63.1 25.8 33.3 26.4 Table 2: Web testing performance of representative LLM on WebTestBench under the WebTester framework. We report the Coverage metric (Cov.) for checklist generation, and the average number of iteration turns (#Turns), average context tokens (#Tokens) per instance, and Precision/Recall/F1 metrics for defect detection. Results in bold and underline denote the best and second-best performances.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

updates are easily mistaken as functional failures, reflecting insufficient model understanding of dynamic web behavior. On the other hand, the low recall poses a greater risk. Most models fail to exceed a 25% recall, meaning the major real defects remain undetected. Beyond limited defect cognition, these false-negatives are partly from a defaultcorrectness bias, where models default to a pass judgment when no explicit evidence is observed. Additionally, we observe a strategic divergence in CUA behavior: they either employ an aggressive detection strategy that favors recall at the cost of precision (e.g., GPT-5.1 achieves 33.3% recall but only 25.8% precision) or adopt a conservative one that prioritizes precision while overlooking numerous real defects (e.g., MiMo-V2-Flash achieves 34.8% precision but only 24.6% recall). Long-horizon Interaction Unreliability. Completing a comprehensive web defect detection process typically requires dozens of interaction turns and millions of tokens. For example, Step-3.5Flash requires an average of 57.0 turns and 3.37M tokens per sample. Such long-horizon tasks demand rigorous long-context memory and planning stability. Specifically, as the interaction history accumulates, models become increasingly susceptible to tracking failures, resulting in the loss of prior states or the execution of redundant operations.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

ing agents requires environments that are ecologically valid and contain diverse and non-trivial defects. However, standard web resources often present limitations. Commercial websites are typically well tested and continuously updated, which makes them unsuitable as benchmark samples. Open-source projects often feature simple designs with shallow structures or limited interactive functionality, and therefore fail to reflect realistic user interactions. To bridge this gap, we utilize Lovable.dev1, an AI-powered web development platform that generates complete websites from user instructions, to synthesize web application projects. Through this process, we obtain an initial web application for each instruction, providing a realistic webpages for defect detection. Gold Checklist and Result Annotation. Given an development instruction and its application, human annotators construct a testable checklist. Inspired by software quality and evaluation standards (e.g., ISO/IEC 25010 (ISO/IEC, 2023)), and adapting

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Annotators derive a checklist from the instruction, and then interact with the actual application to align test cases with the implemented components and interaction flows. Finally, annotators execute the checklist by interacting with the website and document the Pass/Fail status of each item and provide concise bug reports for failures. Iterative Refinement. Initial synthesis often produces applications with few defects, which limits the effectiveness in discriminatively evaluating web testing capabilities. To address this, annotators perform a iterative refinement. This involves revising the instruction for re-generate app or continuing the conversation with lovable.dev to add new features. Throughout this iteration, the checklist and results are updated synchronously until the samples contain sufficient defects for evaluation. Quality Control. To ensure the quality of the dataset, all annotators undergo related training and conduct cross-validation during the annotation process. Finally, a senior annotation leader (nonauthors) performs a final scan of the entire dataset, providing feedback and guiding annotators to op-

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Total items 1750 Functionality 854 Constraint 398 Interaction 247 Content 251 Total Pass / Fail items 1302/448 Functionality (Pass / Fail) 653/201 Constraint (Pass / Fail) 270/128 Interaction (Pass / Fail) 176/71 Content (Pass / Fail) 203/48

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

The term “test oracle” first appeared in William Howden’s seminal work in 1978 [99].

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

In this research, we present the first comprehensive study to investigate the capabilities of LLMs in generating correct, diverse, and strong test oracles capable of effectively identifying a large number of unique bugs. To this end, we fine-tuned seven code LLMs using six distinct prompts on a large dataset consisting of 110 Java projects. Utilizing the most effective finetuned LLM and prompt pair, we introduce TOGLL, a novel LLM-based method for test oracle generation. To investigate the generalizability of TOGLL, we conduct studies on 25 unseen large-scale Java projects. Besides assessing the correctness, we also assess the diversity and strength of the generated oracles. We compare the results against EvoSuite and the state-of-the-art neural method, TOGA. Our findings reveal that TOGLL can produce 3.8 times more correct assertion oracles and 4.9 times more exception oracles than TOGA. Regarding bug detection effectiveness, TOGLL can detect 1,023 unique mutants that EvoSuite cannot, which is ten times more than what TOGA can detect. Additionally, TOGLL significantly outperforms TOGA in detecting real bugs from the Defects4J dataset.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

RQ2 Findings: TOGLL generates significantly more correct test oracles than TOGA; bettering it by 3.8 times and 4.9 times for assertion oracles and exception oracles, respectively.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

In this paper, we investigate what actually drives exceptionoracle prediction in neural and LLM-based TOG. We answer this through a large-scale intervention-based study of three representative TOG systems on three real-world Java datasets: two generatedtest benchmarks (OE25 and Sf110) and OE25dev, a newly curated benchmark of developer-written tests from 25 systems. We define exception-oracle accuracy as an oracle-type prediction metric: for an exception-labeled instance, a prediction is correct when the instance is classified as Exception, not Assertion. This metric does not measure semantic equivalence, compilability, oracle style, or exact exception-type matching. Section 2.4 gives the formal definition. To identify the signals behind these predictions, we compare model behavior before and after removing Javadoc @throws clauses, and then apply attribution-guided substitution ablations with cue categorization across the test prefix, focal code, and documentation. Removing these exception clauses (ECs) causes no change or only very small drops in accuracy: the largest observed overall drop is 0.16 percentage points, and the largest drop among samples that originally contained exceptional-behavior clauses is 0.54 percentage points. This suggests that structured exception documentation is not the main driver. Instead, attribution-guided ablation shows that

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

RQ2 Finding: Javadoc exceptional-behavior clauses provide only a weak auxiliary signal for exception-oracle prediction. Removing ECs causes no change or only small drops in accuracy, with the largest observed Δμbeing -0.16 percentage points and the largest observed ΔECbeing -0.54 percentage points for Doc2OracLL on OE25.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

We investigate this question through a large-scale interventionbased study of three TOG systems spanning classifier-based and generative architectures and model sizes from roughly 110M to 7B parameters, evaluated on three real-world benchmarks comprising two generated-test datasets and a new benchmark of developerwritten tests. We first remove Javadoc @throws clauses and find that accuracy changes only marginally, with the largest drop below one percentage point. This indicates that structured exception documentation is not the primary driver of exception-oracle prediction. We then apply attribution-guided substitution ablations to identify the signals that predictions depend on. The results show that high accuracy can be driven by shortcut signals: some models are highly sensitive to a small number of structural tokens, while others distribute reliance across many lexical cues.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

To address this challenge, we present AugmenTest, an approach leveraging Large Language Models (LLMs) to infer correct test oracles based on available documentation of the software under test. Unlike most existing methods that rely on code, AugmenTest utilizes the semantic capabilities of LLMs to infer the intended behavior of a method from documentation and developer comments, without looking at the code. AugmenTest includes four variants: Simple Prompt, Extended Prompt, RAG with a generic prompt (without the context of class or method under test), and RAG with Simple Prompt, each offering different levels of contextual information to the LLMs. To evaluate our work, we selected 142 Java classes and generated multiple mutants for each. We then generated tests from these mutants, focusing only on tests that passed on the mutant but failed on the original class, to ensure that the tests effectively captured bugs. This resulted in 203 unique tests with distinct bugs, which were then used to evaluate AugmenTest. Results show that in the most conservative scenario, AugmenTest’s Extended Prompt consistently outperformed the Simple Prompt, achieving a success rate of 30% for generating correct assertions. In comparison, the state-of-the-art TOGA approach achieved 8.2%. Contrary to our expectations, the RAG-based approaches did not lead to improvements, with performance of 18.2% success rate for the most conservative scenario.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Background. The oracle problem (determining the correct expected outcome for a test) remains a major bottleneck in automated testing, and is increasingly relevant as non-experts rely on AIgenerated code they cannot reliably validate. Objective. We study whether large language models (LLMs) can generate generalizable test oracles directly from natural-language business requirements, without access to source code or example input–output pairs. Method. We propose a reproducible, requirement-driven pipeline grounded in Defects4J. For each of 10 real bugs from Defects4J Lang (Bugs 1 and 3–11), we (i) extract behavioral changes via buggy/fixed diffs, (ii) manually translate the change into a business requirement, (iii) construct a requirement-derived oracle (REQ) as a gold standard, and (iv) prompt five LLMs (DeepSeek-V3, Gemma-3n, Llama-3, Mistral-7B, and Qwen-3) to generate Java oracle code. We evaluate oracle correctness and generalization under two targets: agreement with REQ and agreement with the system under test (SUT), reporting macro-averaged accuracy, precision, recall, and F1. Results. LLMs achieve non-trivial generalization but with substantial bug- and model-level variance. Generated oracles align more closely with REQ than with SUT, and correlations between requirement technicality/ambiguity ratings and oracle accuracy are weak with wide confidence intervals. Conclusion. No detectable linear relationship exists between requirement properties and oracle accuracy in this dataset, suggesting that pretraining coverage and the semantic specificity of the required behavior dominate oracle correctness. As a pilot proof of concept, these findings are preliminary and are intended to establish feasibility and motivate larger-scale empirical investigation.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

When combined with the example code (including comments), LLMs can generate test cases that achieve high line and branch coverage. However, as demonstrated in prior work [17], [20], [21], high coverage does not necessarily imply strong fault-detection capability when measured by the mutation score. For instance, in our experiments, LLMs generate tests for the subject id_81 from HumanEval-Java

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

with 100% line and branch coverage, yet the corresponding mutation score is only 4%.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

To better understand the role of agent-written tests, we analyze trajectories produced by six strong LLMs on SWE-bench Verified. Our results show that test writing is common, but resolved and unresolved tasks within the same model exhibit similar test-writing frequencies. When tests are written, they mainly serve as observational feedback channels, with value-revealing print statements appearing much more often than assertion-based checks. Based on these insights, we perform a prompt-intervention study by revising the prompts used with four models to either increase or reduce test writing. The results suggest that prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes in this setting. Taken together, these results suggest that current agent-written testing practices reshape process and cost more than final task outcomes.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Abstract—Generation of thorough test oracles is an open problem. Popular test case generators, like EvoSuite and Randoop, rely on implicit, rule-based, and regression oracles that miss failures that depend on the semantics of the program under test. Formal specifications can yield test oracles but are expensive to create. Large Language Models (LLMs) have the potential to overcome these limitations. The few studies of using LLMs to generate test oracles use modest-sized public benchmarks, such as Defects4J, that are likely to be included in the LLM training data, which threatens the validity of the results. This paper presents an empirical study of the effectiveness of LLMs in generating test oracles. Our experiments use 13,866 test oracles, from 135 Java projects, that were created after the LLMs training cutoff dates. Thus, our dataset is unbiased. In our experiments, LLMs generated oracles with average mutation score of 43% — similar to the 45% score of human-designed test oracles. Our results also indicate that the test prefix and the methods called in the program under test provide sufficient information to generate good oracles, while additional code context does not bring relevant benefits. These findings provide actionable insights into using LLMs for automatic testing and highlight their current limitations in generating complex oracles.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Previous work evaluates LLMs on modest-sized public datasets [18], such as Defects4J [27]. This well-known benchmark is likely to have been included in LLM training data; such data leakage may inflate performance estimates [28]– [30]. A recent survey [31] confirms the absence of large-scale evaluation to assess the effectiveness of LLMs in generating assertions for test cases added after the LLM’s training cutoff. This highlights a critical gap in understanding the true generalization ability of LLMs for generating oracles [32]. This paper fills the gap with a large-scale empirical study of concrete test oracles generated by LLMs on a dataset designed to avoid leakage from the training set. We extracted 13,866 oracles from 135 open-source Java projects. All of these test cases were created after 2024-09-01, ensuring that the test code was not in the models’ training data. We evaluated 10 LLMs from 3 families (llama, phi, and qwen), including general-purpose, code-specific, and reasoning-enhanced variants, with 4 different prompt configurations. Our experiments generated 610,104 oracles: 13,866 oracles per LLM-andprompt pair.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

The resulting natural language specifications can also be executed to reveal correct and problematic parts of a software.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

These research questions are operationalized by explicitly mapping them to the survey’s evidence base. RQ1 (techniques) is addressed in Sections IV and V through the corpus of twenty-six primary studies organized across the three evolutionary eras. RQ2 (tools and frameworks) is addressed in Section VI, where Table III summarizes related work and provides a cross-cutting comparison of tools. RQ3 (evaluation) is addressed in Section VI through Table IV, particularly criterion K5 on evaluation thoroughness. RQ4 (gaps and challenges) is addressed in Section VII through four formalized research gaps: G1 (hallucination), G2 (traceability), G3 (complexity sensitivity), and G4 (compliance). Each gap is grounded in specific cell values in Table IV and in external quantitative baselines from [16] and [53] and is mapped one-to-one in Fig. 7 to the four actionable recommendations (R1-R4) developed in Section VIII. This explicit chain from research question to section, table, gap, and recommendation ensures that each finding remains traceable to its evidentiary basis.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

it is simpler to reason about relations between outputs of a program, than it is to fully understand or formalise its input-output behaviour.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Abstract—A natural method to evaluate the effectiveness of a testing technique is to measure the defect detection rate when applying the created test cases. Here, real or artificial software defects can be injected into the source code of software. For a more extensive evaluation, injection of artificial defects is usually needed and can be performed via mutation testing using code mutation operators. However, to simulate complex defects arising from a misunderstanding of design specifications, mutation testing might reach its limit in some cases. In this paper, we present an open-source benchmark testbed application that employs a complement method of artificial defect injection. The application is compiled after artificial defects are injected into its source code from predefined building blocks. The majority of the functions and user interface elements are covered by creating front-end-based automated test cases that can be used in experiments.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

In contrast to the established classical code mutation operators, various complex software defects can be introduced into the code, especially defects caused by a misunderstanding of the SUT design specification or requirements during the development process. The practical use case of the presented testbed is to provide researchers with a complementary option to the mutation testing technique to be able to simulate a broader spectrum of possible software defects during experiments. The testbed is, hence, a complement to mutation testing rather a replacement of mutation testing via a defect injection approach. As we show later in Section II, both approaches have certain advantages and disadvantages. Hence, both approaches can be combined to provide the best objective measurement of the effectiveness of a testing technique.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Concern whether the introduced defects represent typical defects that are being made during real software projects can be raised. This responsibility in experiments is up to the researchers and testing practitioners. Typical defects might vary between various software architectures, development styles, programming languages, business domains, and even decades when the empirical observations are made. Hence, the testbed provides a general possibility to create different types of defects and defect clones, and the decision is up to the testbed user. In the proposed concept, the artificial defects are selected from a pre-defined set, which might limit the generalization of experiment results. This potential limit can be solved by the addition of more artificial defects as well as the correct interpretation of the results of the experiments.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

JavaScript is a popular programming language that is also error‐prone due to its asynchronous, dynamic, and loosely typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BUGSJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server‐side programs, comprising 444k lines of code (LOC) in total. Each bug is accompanied by its bug report, the test cases that expose it, as well as the patch that fixes it. We extended BUGSJS with a rich web interface for visualizing and dissecting the bugs’ information, as well as a programmable API to access the faulty and fixed versions of the programs and to execute the corresponding test cases, which facilitates conducting highly reproducible empirical studies and comparisons of JavaScript analysis and testing tools. Moreover, following a rigorous procedure, we performed a classification of the bugs according to their nature. Our internal validation shows that our taxonomy is adequate for characterizing the bugs in BUGSJS. We discuss several ways in which the resulting taxonomy and the benchmark can help direct researchers interested in automated testing of JavaScript applications. © 2021 The Authors. Software Testing, Verification & Reliability published by John Wiley & Sons, Ltd.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Agent systems rely on LLM APIs for every response, but these APIs can return server errors, truncated responses, or corrupted content that propagates through downstream agents and causes task failure. Evaluating robustness under these faults is crucial for reliable deployment. Existing fault injection methods are offline, require source code modification, or cannot modify specific response fields. A comprehensive evaluation also requires a systematic fault taxonomy because different fault types affect downstream agents differently. We propose AgentChaos, a chaos engineering framework for controlled, runtime, non-intrusive LLM API fault injection. Since all agent systems access LLMs through the same HTTP interface, we inject faults at this shared layer without modifying source code. We define crash, omission, and value faults on content and tool call fields, intercept and modify LLM API responses at runtime, and verify whether each fault is triggered to filter untriggered tasks and avoid underestimating fault impact. Evaluations across agent

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

4.4 Trigger Verification An agent system decides how many LLM calls to make at runtime, and this number varies across tasks. A fault configured for a specific call position may not fire if the task finishes before reaching that position. For example, if a fault is configured for the 3rd LLM call but the task finishes in 2 calls, the fault is never applied. The intermittent strategy may also not select any call during a short task. The target field guard (§4.3) may cause additional skips if the LLM consistently returns tool calls when a content fault is configured. Including such untriggered tasks in the evaluation would mix faulted and unfaulted results, making the system appear more robust than it actually is. We therefore check after each task whether the execution trace contains at least one fault event (as recorded by the wrapper in

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

§4.3). If no fault event exists, the task is marked as untriggered and excluded from the evaluation. Only triggered tasks are used to compute pass@1w/ FI and Δpass@1. This filtering ensures that the measured degradation reflects the true impact of the injected fault.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

provide useful signals when comparing across models. However, in the more general and practically challenging setting where the correctness of the code-under-test provided to the LLM cannot be guaranteed, and the generated tests are expected to detect bugs present in that code, coverage becomes unreliable as an indicator of bug detection effectiveness, and mutation analysis is not applicable. Furthermore, unlike both prior studies, we find little evidence that the size of LLMgenerated test suites (i.e., the number of tests) is a strong confounding factor in the relationships among coverage, mutation score, and real-bug detection effectiveness.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

1We do not conduct mutation testing, or analyze the correlation between mutation score and bug detection, for buggy code in RQ2 because mutation testing presupposes a passing (“green”) test suite on the code-under-test; this would exclude precisely the tests that expose the bug within the code-under-test, hiding the very bug to be detected and rendering the resulting mutation scores meaningless. Mutation analysis is therefore not applicable in this setting.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Given that Large Language Models (LLMs) are increasingly applied to automate software development, comprehensive software assurance spans three distinct goals: regression prevention, reactive reproduction, and proactive discovery. Current evaluations systematically overlook the third goal. Specifically, they either treat existing code as ground truth (a compliance trap) for regression prevention, or depend on post-failure artifacts (e.g., issue reports) for bug reproduction—so they rarely surface defects before failures. To bridge this gap, we present TestExplora, a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments. TestExplora contains 2,389 tasks from 482 repositories and hides all defect-related signals. Models must proactively find bugs by comparing implementations against documentationderived intent, using documentation as the oracle. Furthermore, to keep evaluation sustainable and reduce leakage, we propose continuous, timeaware data collection. Our evaluation reveals a significant capability gap: state-of-the-art models achieve a maximum Fail-to-Pass (F2P) rate of only 16.06%. Further analysis indicates that navigating complex cross-module interactions and leveraging agentic exploration are critical to advancing LLMs toward autonomous software quality assurance. Consistent with this, SWEAgent instantiated with GPT-5-mini achieves an F2P of 17.27% and an F2P@5 of 29.7%, highlighting the effectiveness and promise of agentic exploration in proactive bug discovery tasks.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

We conduct a comprehensive evaluation utilizing four key metrics: Head Pass Rate, Fail-to-Pass Rate, Entry Coverage, and Change-focused Coverage. Additionally, the number of testcases (Num.) is reported. Our experiments involve six representative language models evaluated on 12,227 real-world pull requests. To ensure evaluation efficiency, we constructed a high-quality subset, TestExplora-Lite, by filtering samples based on the quality of human-written docstrings. This subset comprises 330 PRs and 517 samples in total. All baseline experiments are repeated three times to ensure the robustness of the results.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exactmatch agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MTBench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen’s κ is universal (33–41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test–retest reliability (> 0.95) coexists with severe position bias (> 0.10) in two production-deployed judges (instantiating a consistency–bias paradox), and verbosity bias is small (< 0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

We report five principal findings, including two diagnostic concepts. First, kappa deflation: raw agreement overstates chance-corrected discrimination by 33–41pp in all 21 evaluated models. Second, judge rankings are non-transferable: models shift by as many as 14 positions across benchmarks. Third, the consistency–bias paradox: high test– retest reliability often masks severe position bias. Fourth, verbosity bias is much reduced: all 21 models register <0.011, in sharp contrast to the 20– 40% variance reported in 2023 literature. Fifth, JudgeBench discriminates 4.5× more sharply than MT-Bench (60.4pp vs. 13.5pp κ spread).

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Pearson’s r) to evaluate judge quality, we demonstrate that correlation alone is insufficient. Our methodology progresses to Cohen’s Kappa, which measures actual agreement rather than just linear relationships. This addresses critical issues like systematic bias—an LLM could have perfect correlation while consistently being too harsh or lenient. Second, we design a novel Turing Test for judges based on Cohen’s Kappa agreement patterns. Unlike traditional Turing Tests that focus on conversational indistinguishability, our approach asks: “When mixed with human annotators, can we distinguish the LLM from typical human judges?” This test uses z-score analysis of Cohen’s Kappa values to identify models that judge like typical human annotators (|z| < 1) versus those with exceptional consistency patterns. These innovations establish a more rigorous framework for validating LLM judgment excellence, moving beyond superficial correlation to actual functional capability, forming a new benchmark—the Judge’s Verdict Benchmark—that provides a standardized way to assess whether an LLM achieves Tier 1 performance for either human-like evaluation or maximum-consistency tasks.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

To assess the alignment of each LLM-as-a-judge, we used the Answer Accuracy metric from the RAGAS1 library which measures how closely a generated answer matches a reference answer. It employs a Large Language Model (LLM) as an evaluator, or “judge,” to assess the factual and semantic concordance between the reference and the generated response. The process involves two independent LLM-as-a-judge prompts, each of which are prompted with the user’s question, the system-generated answer, and the reference answer in different orders. Each judge assigns a discrete score of: 0 (No Alignment), 2 (Partial Alignment), or 4 (Exact Alignment) 1. To derive the final score, these discrete ratings are first normalized to a continuous scale in 2. The normalized scores of the two judges are then averaged to produce a final accuracy score of the generated answer in 3. This diverse method improves the reliability of the evaluation by mitigating the positional bias and increases the robustness of a single LLM-as-a-judge prompt. More formally, let Si ∈{0, 2, 4} denote the discrete score of the LLM judge i comparing the reference and the generated response, where:

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM’s ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https: //github.com/RANDCorporation/judge-reliability-harness

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。

Judge output robustness is highly task-dependent. Models that appear stable in binary safetyclassification settings (e.g., FORTRESS or HarmBench) degrade substantially when required to assign multi-level ordinal scores, as in Persuade. Practitioners who rely on ordinal scoring or preference-ranking tasks may be overestimating the reliability of their evaluation systems. Formatting perturbations produce larger reliability drops than semantic perturbations. This asymmetry is concerning, as different LLMs tend to have unique quirks in how they format their responses. Judges that are brittle to such differences risk embedding instability into downstream model comparisons or leaderboard decisions, even when semantic content remains unchanged.

文字由 PDF 擷取並合併換行；字形與版面以截圖為準。