# Ablation Study Commands (v2) **Dataset:** QALD9Plus Train (348 questions) **Primary metric:** Macro-F1 on QALD answers **Entities:** Gold (held fixed across all runs) **Carried-forward models (Phase 0 assumption):** QWEN3D6_27B, GPTOSS120B --- ## Placeholder Legend | Placeholder | Meaning | |---|---| | `[BEST_LLM]` | Winning model from Phase 1 (QWEN3D6_27B or GPTOSS120B) | | `[N*]` | Winning Top-N from Phase 1 | | `[M*]` | Winning M-Hop from Phase 1 | | `[SCHEMA_FLAGS]` | Winning AT/CI/CE combination flags from Phase 2 | | `[WINNER_FLAGS]` | All winning feature flags (schema + PC + EV) combined | | `[NEIGHBOR_*]` | Top-N neighbors of N* plus extremes 5 and 500 | | `[SEED_A]`, `[SEED_B]` | Two additional random seeds for Phase 5 | **Base flags used in every command:** ``` --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --language en --use-gold ``` --- ## Phase 0: Model Selection (6 runs) Already executed. See strategy document for commands. --- ## Phase 1: Top-N x M-Hop Interaction Grid (42 runs) All schema/pattern/verification features = n. ### QWEN3D6_27B (21 cells) ```bash # M-Hop = 1 (all Top-N) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 5 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 10 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 20 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 50 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 100 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 500 --mhop-limit 1 --language en --use-gold # M-Hop = 2 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 5 --mhop-limit 2 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 20 --mhop-limit 2 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 100 --mhop-limit 2 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 500 --mhop-limit 2 --language en --use-gold # M-Hop = 5 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 20 --mhop-limit 5 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 50 --mhop-limit 5 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 100 --mhop-limit 5 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 500 --mhop-limit 5 --language en --use-gold # M-Hop = 10 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 50 --mhop-limit 10 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 100 --mhop-limit 10 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 500 --mhop-limit 10 --language en --use-gold # M-Hop = 20 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 100 --mhop-limit 20 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 500 --mhop-limit 20 --language en --use-gold # M-Hop = 50 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 100 --mhop-limit 50 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count 500 --mhop-limit 50 --language en --use-gold ``` ### GPTOSS120B (21 cells) ```bash # M-Hop = 1 (all Top-N) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 5 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 10 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 20 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 50 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 100 --mhop-limit 1 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 500 --mhop-limit 1 --language en --use-gold # M-Hop = 2 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 5 --mhop-limit 2 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 20 --mhop-limit 2 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 100 --mhop-limit 2 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 500 --mhop-limit 2 --language en --use-gold # M-Hop = 5 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 20 --mhop-limit 5 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 50 --mhop-limit 5 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 100 --mhop-limit 5 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 500 --mhop-limit 5 --language en --use-gold # M-Hop = 10 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 50 --mhop-limit 10 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 100 --mhop-limit 10 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 500 --mhop-limit 10 --language en --use-gold # M-Hop = 20 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 100 --mhop-limit 20 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 500 --mhop-limit 20 --language en --use-gold # M-Hop = 50 bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 100 --mhop-limit 50 --language en --use-gold bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count 500 --mhop-limit 50 --language en --use-gold ``` --- ## Phase 2: Schema Feature Full Factorial (7 runs) Fixed: best model, M*, N*, PC=n, EV=n. The (n, n, n) cell was already covered in Phase 1. Replace `[BEST_LLM]`, `[N*]`, `[M*]` with Phase 1 winning values. ```bash # AT only bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity # CI only bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-class-info # CE only bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --conc-ex-limit 10 # AT + CI bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --use-class-info # AT + CE bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --conc-ex-limit 10 # CI + CE bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-class-info --conc-ex-limit 10 # AT + CI + CE bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --use-class-info --conc-ex-limit 10 ``` --- ## Phase 3: PC x EV Factorial (3 runs) Fixed: best model, M*, N*, AT*, CI*, CE*. The (n, n) cell was already covered in Phase 2. Replace `[BEST_LLM]`, `[N*]`, `[M*]`, `[SCHEMA_FLAGS]` with winning values from Phase 2. ```bash # PC only bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold [SCHEMA_FLAGS] --include-pattern-count # EV only bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold [SCHEMA_FLAGS] --verify-update-sparql # PC + EV bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold [SCHEMA_FLAGS] --include-pattern-count --verify-update-sparql ``` --- ## Phase 4: Post-Hoc Sensitivity Validation (~18 runs) Fixed: M*, N*, AT*, CI*, CE*, PC*, EV* (candidate winner from Phase 3). ### 4.1 — Model Validation (3 runs) Run all three models at the final config to validate Phase 0 model choice. ```bash bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm QWEN3D6_27B --topn-count [N*] --mhop-limit [M*] --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GPTOSS120B --topn-count [N*] --mhop-limit [M*] --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm GEMMA4_31B --topn-count [N*] --mhop-limit [M*] --language en --use-gold [WINNER_FLAGS] ``` ### 4.2 — Top-N Neighbors (~3-4 runs) Run Top-N values adjacent to N*, plus extremes 5 and 500 if not already neighbors. ```bash bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [NEIGHBOR_1] --mhop-limit [M*] --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [NEIGHBOR_2] --mhop-limit [M*] --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [NEIGHBOR_3] --mhop-limit [M*] --language en --use-gold [WINNER_FLAGS] ``` ### 4.3 — M-Hop Full Sweep (5 runs) Run all M-Hop values except M* (already done). ```bash bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit 1 --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit 2 --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit 5 --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit 10 --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit 20 --language en --use-gold [WINNER_FLAGS] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit 50 --language en --use-gold [WINNER_FLAGS] ``` > **Note:** Remove the line where `--mhop-limit` equals `[M*]` since it was already run. ### 4.4 — Leave-One-Out Feature Abllation (up to 5 runs) For each feature that is "y" in the winner, flip it to "n". Only include lines for features actually enabled in the winning config. ```bash # AT flipped off (remove --use-aug-similarity from winner) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-class-info --conc-ex-limit 10 --include-pattern-count --verify-update-sparql # CI flipped off (remove --use-class-info from winner) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --conc-ex-limit 10 --include-pattern-count --verify-update-sparql # CE flipped off (remove --conc-ex-limit 10 from winner) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --use-class-info --include-pattern-count --verify-update-sparql # PC flipped off (remove --include-pattern-count from winner) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --use-class-info --conc-ex-limit 10 --verify-update-sparql # EV flipped off (remove --verify-update-sparql from winner) bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [BEST_LLM] --topn-count [N*] --mhop-limit [M*] --language en --use-gold --use-aug-similarity --use-class-info --conc-ex-limit 10 --include-pattern-count ``` > **Note:** Only run the lines corresponding to features that are actually "y" in the winning configuration. --- ## Phase 5: Seed Replication (6 runs) Top 3 configurations across the entire study, each run with 2 additional seeds. ```bash # Top config #1 (winner) x 2 seeds bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [LLM_1] --topn-count [N_1] --mhop-limit [M_1] --language en --use-gold [FLAGS_1] --seed [SEED_A] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [LLM_1] --topn-count [N_1] --mhop-limit [M_1] --language en --use-gold [FLAGS_1] --seed [SEED_B] # Top config #2 x 2 seeds bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [LLM_2] --topn-count [N_2] --mhop-limit [M_2] --language en --use-gold [FLAGS_2] --seed [SEED_A] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [LLM_2] --topn-count [N_2] --mhop-limit [M_2] --language en --use-gold [FLAGS_2] --seed [SEED_B] # Top config #3 x 2 seeds bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [LLM_3] --topn-count [N_3] --mhop-limit [M_3] --language en --use-gold [FLAGS_3] --seed [SEED_A] bash slurm/schedule_experiment.sh --approach PBSG_MHOP --dataset QALD10_UPDATED_TENTRISMAIN --split TRAIN --llm [LLM_3] --topn-count [N_3] --mhop-limit [M_3] --language en --use-gold [FLAGS_3] --seed [SEED_B] ``` --- ## Run Budget Summary | Phase | Focus | Runs | |---|---|---| | 0 | Model selection + cross-check | 6 | | 1 | Top-N x M-Hop interaction grid | 42 | | 2 | Schema feature full factorial (AT x CI x CE) | 7 | | 3 | PC x EV factorial | 3 | | 4 | Post-hoc sensitivity validation | ~18 | | 5 | Seed replication on top 3 | 6 | | **Total** | | **~82** | --- ## Flag Reference | Flag | Feature | Effect | |---|---|---| | `--use-aug-similarity` | AT (Augmented Text for similarity) | Use augmented text in similarity computation | | `--use-class-info` | CI (Class/Domain/Range Info) | Include class information in prompt | | `--conc-ex-limit 10` | CE (Concrete Examples) | Include up to 10 concrete examples | | `--include-pattern-count` | PC (Pattern Count) | Include pattern count in prompt | | `--verify-update-sparql` | EV (Extra Verification Update) | Run extra SPARQL verification step |