Learning and Benchmarking Process Logic in Semiconductor Fabrication Routes

Team Attention Seekers. Kyrillus Mehanni (Senior Software Engineer, symbolic approach and DevOps), Julian Schmidt (AI Engineer, model training), Emil Kascper (AI Engineer, model training), and Abdul Basit Banbhan (AI Researcher and Engineer, direction of AI research and model training). Track: Industrial AI (Infineon). This page is a replica of the submission report, with every illustration included. The source of truth is SUBMISSION.md in the repository, and the figures are produced by shared/benchmark/report.py from the official scorer output.

Summary

We built three process logic systems for semiconductor fabrication routes, a decoder transformer, a self supervised hybrid, and a zero parameter neurosymbolic engine, and scored all three, plus two baselines, on one shared labeled eval set with the official eval_metrics.py, both in distribution and on a held out product family. The central finding is that in distribution accuracy is a trap: a 50 line trigram already reaches Top 5 of about 0.99, so it does not separate approaches, and bigger models do not help either, since three transformer sizes converge to within 0.0001 language model loss. What separates memorization from process logic is generalization to an unseen family and whether completions respect the rules: the trigram introduces a new rule violation in 50 percent of its completions, while every structure aware system stays at zero new violations. Our single largest improvement was not a bigger model but fixing a context window bug that was truncating the process backbone, which raised held out next step accuracy by 29 points.

Problem

Semiconductor routes are ordered sequences of roughly 110 to 150 steps over a vocabulary of about 120 step strings, where validity is defined by ten documented forbidden patterns, for example, a deposition requires a prior clean, and electrical tests must follow passivation.

Before training anything we computed a trigram with backoff. It reaches Top 5 of about 0.99 in distribution, identical to the memorization upper bound, and its in distribution accuracy is unchanged on a held out split. The provided task therefore carries almost no model relevant entropy in distribution. The same trigram collapses out of distribution: under a leave one family out split its next step accuracy drops sharply, which quantifies the real gap. So the problem we chose is not raw in distribution accuracy but two harder questions: does a system generalize its process understanding to a product family it never saw in training, and does it complete and validate routes in a way that respects process logic rather than reproducing memorized strings. The work targets the three scored tasks, next step prediction, sequence completion, and anomaly detection with rule attribution, and the organizer evaluated fourth task, out of distribution generalization to a hidden family, approximated here by a leave one family out protocol.

Approach

• One eval set, one official scorer, five systems, two regimes. Every system is scored on the same frozen labeled eval set with eval_metrics.py, in distribution, where all three families are in training, and leave one family out, where two families are in training and the held out third is scored. This makes the comparison like for like, which the earlier per model reports did not.
• Decoder transformer with an xLSTM variant, the submission model. Rotary position embeddings, RMSNorm, multitask validity and rule heads, trained on an online generator stream with 40 percent of sequences carrying a labeled rule violation. A compositional tokenizer splits each step string into word tokens, so a new step in an unseen family that shares words is not fully out of distribution. The dominant quality lever is the context window, because a short window truncates the long route.
• Self supervised hybrid with retrieval. A self supervised causal model with semantic step features and product family embeddings, plus a learned contrastive reranker over its Top 5 candidates.
• Neurosymbolic engine with zero trained parameters. An induced grammar and the ten rule oracle define which next steps are legal at each prefix; a role factored Prediction by Partial Matching ranker orders the legal candidates. Correctness is owned by the symbolic spine, so it transfers to an unseen family with almost no drop.
• Two baselines. A trigram with backoff, the memorization floor, and the same trigram under a grammar mask. The full benchmark runs from a clean checkout on CPU or CUDA; production grids run on the Leonardo cluster (NVIDIA A100).

How to run it

The whole comparison reproduces from a clean checkout, on CPU or CUDA, with no cluster access.

reproduce.sh runs five stages under shared/benchmark/: build the frozen eval set with deterministic seeds (make_eval_set.py), generate training data (make_train_data.py), train the compact checkpoints (train_txl.sh, train_ssl.sh), run every system and score with the official metrics (make_benchmark.py), then aggregate tables and figures (report.py). It auto detects CUDA and otherwise runs on CPU; the Apple MPS backend is skipped because its embedding gather kernel is unstable for this model.

On Leonardo the same pipeline runs on one A100 in about 15 minutes. Run from a login node; it submits to a GPU node and needs no internet on the compute node.

reproduce.sbatch requests one A100 on the boost_usr_prod partition, loads the cuda and gcc modules, and runs reproduce.sh on the GPU using the environment from step 2. Add --reservation=<name> only if your project uses one. Outputs match the local run: shared/benchmark/results_summary.csv and submission/UNIFIED_BENCHMARK.md. The full scale training grids, context window 768, the xLSTM architecture, and multiple sizes, are in shared/scripts/slurm/.

Score one task directly with the official scorer, in either environment.

Results

The eval set is built by make_eval_set.py with benchmark only seeds (90001, 90002, 90003 valid; 90010, 90011 invalid), disjoint from training. Next step is scored at the organizer 60 percent and 80 percent truncation points.

Headline metric. In distribution next step Top 1: the transformer leads at 0.779, the neurosymbolic engine reaches 0.761 with zero trained parameters, and the trigram floor is 0.721.

Task 1, next step prediction

Drop is the in distribution value minus the leave one family out value.

The trigram already reaches Top 5 of 1.000 in distribution, and every learned system lands within about six points on Top 1. In distribution next step does not separate approaches. The drops are small for all systems under this protocol because the 60 and 80 percent cut points fall in the process back end, metallization, passivation, test, and ship, which is shared across families; an all position protocol that also probes family specific front end positions drives the trigram down toward 0.47.

Task 2, sequence completion

Edit distance is lower is better. Rule clean is the fraction of completions that introduce no new rule violation beyond the prefix.

The transformer is closest to the reference, with edit distance 0.242. The decisive column is rule clean: the trigram violates a process rule in half of its completions, 0.500, while every structure aware system stays at 1.000 in distribution. The neurosymbolic edit distance is high, 0.706, by design, because it emits a guaranteed rule valid but length divergent completion. Exact match is 0.000 for all systems and is not the right yardstick for a 30 to 60 step suffix.

Task 3, anomaly detection and rule attribution

Baselines have no anomaly capability and are omitted.

All three reach F1 of 1.000 in distribution because they share the same symbolic validator, which is exact on the ten rules. The differentiator is rule attribution, where the neurosymbolic engine reaches 1.000, and that all three hold F1 of 1.000 out of distribution. The honest reading of this 1.000 is in the note on honesty below.

Scaling: bigger is not better on this task

A seven cell grid varies architecture, size, and tokenization. Three transformer sizes spanning 4 M to 113 M parameters converge to within 0.0001 language model loss, so capacity is not the bottleneck. The xLSTM variant closes the gap only with scale, never beats the transformer, and runs three to four times slower per step. Step as token and compositional losses are not directly comparable because their per token entropy differs.

Per family breakdown

The leave one family out columns above are the out of distribution view; the figure below shows next step Top 1 per held out family.

Every value in the task tables is produced by the official scorer through shared/benchmark/score.py, aggregated by report.py into shared/benchmark/results_summary.csv and submission/benchmark_assets/tables.md, and rendered in submission/UNIFIED_BENCHMARK.md. The scaling and context window numbers come from the production transformer grids logged under shared/extras/. Raw predictions are under shared/benchmark/predictions/<regime>/<system>/.

What worked

Finding and fixing the context window bug. An audit found max_len set to 256 while compositional sequences run 444 to 604 tokens, so the full training set was being truncated, hiding the prefix, clean, prep, and cycle backbone the model was meant to learn. Raising the window from 256 to 768 was the single largest improvement of the project.

• The controlled benchmark. One frozen labeled eval set and one official scorer turned five incomparable per model claims into a defensible head to head result in both regimes.
• Rule clean as the discriminator. Standard metrics saturate and hide the real question. Rule clean exposes it: the memorizing trigram violates a rule in half of its completions, 0.500, while every structure aware system stays at 1.000.
• A zero parameter system that competes. The neurosymbolic engine reaches 0.761 next step Top 1, perfect anomaly and rule attribution, and almost no out of distribution drop, with no trained weights.

What did not work

• Retrieval augmentation. The prefix overlap between the retrieval bank and the eval prefixes was 0 percent, so it changed nothing and was discarded.
• The heuristic reranker. Reordering the Top 5 with validator and trigram penalties moved Top 1 by about 0.0004 and was superseded by a learned contrastive reranker, which gave a real internal gain, Top 1 from 0.7993 to 0.8044.
• The validity head as a standalone anomaly detector. Out of distribution it was badly miscalibrated: on held out valid MOSFET sequences it produced 36 false positives in 100, and its ROC AUC fell from 1.00 to 0.31. The fix was to make the symbolic validator dominant and only let the head override it at high confidence. The earlier 1.00 figure was misleading because the head was undertrained.
• xLSTM and cluster setup. xLSTM cells failed at first because the compute nodes have a CUDA driver but no compiler for the just in time kernels, fixed by loading gcc and cuda modules in every job script; the conda CUDA resolver also failed on the GPU free login node and was replaced by a pinned PyPI build. xLSTM was then dropped from later grids because it is three to four times slower at identical loss.

Another 36 hours

• Run the production scale checkpoints, context window 768, six thousand steps, A100, through this benchmark, so the learned models are compared against the neurosymbolic engine at full strength rather than at the compact reproducible budget.
• Use the neurosymbolic oracle as a reranker and validity filter over the learned model Top 5 lists, where Top 5 of 1.000 leaves headroom to raise Top 1.
• Wire the parsed physics features, temperature, time, thickness, pressure, energy, and dose per step, into the token embedding; they are parsed but not yet injected, and are the largest unexplored out of distribution lever.
• Train a process reward model that predicts, at each prefix, the probability of completing to a valid route, and use it inside beam search to lift completion exact match.
• Add multi seed confidence intervals and a harder out of distribution probe at family specific front end positions; the benchmark currently uses a single seed and a small eval slice.

Deliverables

Track specific deliverables for Industrial AI (Infineon):

• Eval submission files in organizer format: predictions_nextstep.csv, predictions_completion.csv, predictions_anomaly.csv under competition/participant-files/predictions/, plus per regime and per system files under shared/benchmark/predictions/.
• Training artifacts: per run config and final loss in shared/benchmark/checkpoints/*/summary.json with final.pt, self supervised checkpoints in shared/benchmark/ssl_checkpoints/, production checkpoints and TensorBoard loss curves under shared/extras/.
• Scores from eval_metrics.py on all three tasks with per family breakdown, in submission/UNIFIED_BENCHMARK.md, shared/benchmark/results_summary.csv, and submission/benchmark_assets/.
• Demo shows baseline versus trained output on identical inputs: before and after examples plus this dashboard under www/.

Credits: Python, PyTorch (cu121), NX AI xlstm, einops, OmegaConf, TensorBoard, pandas, NumPy, and matplotlib, in a Pixi managed environment, versions pinned in requirements.txt and pyproject.toml. No pre trained models: the transformer and self supervised hybrid train from scratch, and the neurosymbolic ranker has zero trained parameters. No external APIs. AI coding assistant used during the hackathon: Claude Code. Datasets: synthetic semiconductor sequences for MOSFET, IGBT, and IC, provided by the organizers with the scoring script eval_metrics.py; further valid and invalid sequences generated by the committed rule based generator and validator.

A note on honesty

The two eval paths disagree on transformer versus trigram. The unified benchmark adapter scores the transformer above the trigram, 0.779 against 0.721, in distribution. A separate production decode that assembles step strings by beam searching word tokens scores the trained transformer below the trigram, about 0.60 against 0.72, because beam assembly of multi token steps is less reliable than a direct argmax. We did not reconcile the two paths. The unified benchmark is the controlled like for like comparison; the compositional beam search reflects the heavier production path whose value is out of distribution coverage, not in distribution Top 1.

• Anomaly 1.000 rests on a shared validator, not on a learned model. The validator is the organizers' own code and is an oracle for the ten documented rules in distribution, so every validator backed system scores about 1.0. The learned validity head on its own does not generalize, as the miscalibration above shows. The honest differentiator is rule attribution and out of distribution behavior, where the symbolic role induction is strongest.
• Compact versus production checkpoints. The benchmark checkpoints are trained at a reduced budget on CPU so the comparison reproduces on a laptop, the transformer is the small config at 4.37 M, context window 512. They are not the full Leonardo checkpoints, context window 768, six thousand steps, A100. The purpose here is the controlled comparison, not peak accuracy; because of the context window bug above, earlier numbers are known underestimates.
• Completion edit distance versus validity. Edit distance measures similarity to one reference, so a fully rule valid completion can still diverge from it. Both are reported.
• Small eval and single seed. The shared eval set is intentionally small per family and uses a single seed, so a few percentage points of noise are visible. Multi seed intervals are future work.
• Official accuracy is organizer computed. The organizer eval inputs are unlabeled, so official accuracy can only be computed by the organizers. The numbers here are internal held out scores on our labeled eval set, built from the same generator and protocol. Some submission files were first generated against self simulated eval inputs and will be regenerated against the organizer files at submission. No official score is fabricated.