# Why Is My Tag Missing? Tracing the ATDD Tag Extraction Pipeline

The pre-merge ATDD pipeline scans open GitLab merge requests targeting main and computes the minimal set of Gherkin tags describing the acceptance tests affected by that MR. Those tags drive the downstream test run — so if a tag doesn't appear, the tests it guards don't run.

Which leads to the question this post exists to answer:

"I changed something in an MR, but the tag I expected did not appear in the final output. Why?"

The short answer is that detection is only the first of five stages, and a tag can be dropped at any of them. A tag correctly detected in stage 2 can still vanish in stage 4 or stage 5. Diagnosing this means walking the stages in order rather than jumping to the one you suspect.

The End-to-End Flow

1. An open MR is fetched. It is skipped entirely if its CI is not green, or if it has a real merge conflict with main.
2. The MR's changed files are classified by extension and routed to the matching channel's detector.
3. Each detector appends qualifying tag blocks to a shared tag.txt (the Java channel also feeds java.json).
4. processTagBlock aggregates all blocks into one de-duplicated, or-joined tag string, applying feature-level filters.
5. tag_filter.py intersects that string with the curated EpicTags.txt allow-list and appends @LMSODO* pass-through tags, producing the final tag_filter.txt.

Every change reaches the output through one of three independent channels, decided purely by file extension: .feature, .java, and .xlsx.

Channel 1 — Feature Files (.feature)

Handled by detect_changes.py, which runs four detection paths (A–D) over a git diff --unified=0 of the changed file.

Tags are brought when any of these is true:

• Path A — a scenario's tag line or its Scenario: keyword line was added in the diff (is_new). Source: extract_scenario_tag_groups.
• Path B — a Scenario Outline's Examples table changed, classified as HEADER, DATA_ROW_NUM, or SHEET_NAME. Source: tag_extractor.py:357-436.
• Path C — a feature-level tag line (above Feature:) changed. Emits only the changed feature tags, not every scenario. Source: extract_feature_tags_from_diff.
• Path D — any line added or removed inside a scenario's line span: steps, tables, doc-strings. This catches the edits paths A–C miss. Source: detect_changes.py:225-248.

The change is ignored when:

• The feature carries @UnImplemented — the whole file is skipped immediately (detect_changes.py:277).
• The Scenario Outline or its Examples block is tagged @UnImplemented (tag_extractor.py:363, :369).
• The added line is whitespace-only; indentation and blank-line edits do not count (tag_extractor.py:80-89). This applies separately to Examples header and data-row changes (:405-408, :435-436).
• The scenario's tag set was already emitted by an earlier path — de-duplication (detect_changes.py:243-248).
• The tags are @AuthoredBy-* or @ImplementedBy-* attribution tags. The scenario is kept; these tags are stripped (tag_extractor.py:183-188).

Channel 2 — Java Files (.java)

detect_java_changes.py resolves changed .java files to the method names that changed, writes them to java.json, and an external atdd-cucumber-engine JAR maps those methods to Gherkin tags.

A method is flagged as changed when an added line falls within a HEAD method's span [signature_line, end_line], or a removed line falls within a BASE method's span — so a fully deleted method is still traced, marked (deleted).

The change is ignored when:

• The changed line is an annotation above the signature, outside the method span.
• The line matches a control-flow keyword with a word(...) shape (if, for, while) — not treated as a method.
• The method identity <package> | <signature> was already recorded.
• The method maps to no Gherkin tag in the cucumber-engine JAR. This is the quiet one: detection succeeded, but nothing downstream claims the method.

Channel 3 — Excel Workbooks (.xlsx)

A Scenario Outline can draw its data from an Excel workbook via an Examples: table referencing the workbook filename, often with a sheet name and row number. When an MR changes an .xlsx, three complementary passes run, all appending to tag.txt:

• Pass 1 — the workbook is matched to scenarios strictly by filename as referenced in an Examples table. Source: excel_diff_parser.py → excel_tag_resolver.py.
• Pass 2 — every .feature referencing the workbook, with the Excel column auto-detected by .xlsx/.xls cell values. Source: excel_feature_lookup.py.
• Pass 3 — only scenarios whose Examples row points at a row that actually changed, matched by filename + sheet name + row number. The most precise pass. Source: extract_tags_by_excel_change.py.

The change is ignored when:

• No prior version of the workbook exists — a brand-new file means precise Pass 3 is skipped entirely (Jenkinsfile:1005).
• excel_diff_parser.py exits non-zero — Excel tag resolution for that MR is skipped (Jenkinsfile:943).
• The workbook is not referenced by any .feature Examples table. Filename matching finds nothing, by design.
• The cell change is whitespace-only (tag_extractor.py:76-77).
• A referenced scenario is @UnImplemented or carries attribution tags — filtered like feature tags.

One trap worth internalizing: Passes 1 and 3 assume a fixed workbook layout — row 1 is the title, row 2 the header, data from row 3 (HEADER_ROW_INDEX = 2, _EXCEL_ROW_OFFSET = 2). A workbook not following this layout can silently mis-map rows, so an expected tag may not appear even though the row genuinely changed.

Pipeline-Wide Filters — The Usual Culprits

After all three channels have written to tag.txt, two further stages run once for the whole MR set. These are the most common reason a correctly detected tag disappears.

Aggregation (processTagBlock): a feature block whose feature tags include @UnImplemented is skipped (Jenkinsfile:795); @AuthoredBy-* / @ImplementedBy-* are dropped; and only tokens starting with @ are kept — bare words are discarded (Jenkinsfile:800).

The final allow-list gate (tag_filter.py) is the single biggest "why is my tag missing" cause. A tag survives to tag_filter.txt only if it is either present in the curated EpicTags.txt allow-list, or a @LMSODO* ticket/story tag, which always passes through. Everything else — including a perfectly detected, entirely valid scenario tag — is silently removed (tag_filter.py:158-171).

If the filtered result is empty (no epic match and no @LMSODO tags), tag_filter.py exits with code 2 and the downstream test run is skipped.

MR-Level Skips

Before any tag detection runs at all, an entire MR — and all its scenarios — is skipped when it has no CI pipeline (Jenkinsfile:661), the pipeline is still running (:670) or was canceled (:673), the required java_compilation job was not successful (:202-211, yielding mergeStatus = SKIPPED_CI, DO NOT MERGE), or there is a real merge conflict with main on the MR's own files (:248, :304).

Diagnostic Checklist

Work through these in order. The first "yes" explains the missing tag:

1. Was the whole MR skipped — CI not green, still running, canceled, or a real merge conflict?
2. Did the change land in the channel you assumed, given that routing is purely by file extension?
3. Is the feature, scenario, or Examples block tagged @UnImplemented?
4. Was the edit whitespace-only?
5. For Java: did the line land inside a method body, and does that method map to a tag in the cucumber-engine JAR?
6. For Excel: is the workbook referenced by a .feature, does a prior version exist, and does it follow the expected row layout?
7. Is the tag in EpicTags.txt, or does it start with @LMSODO? If neither, it was silently dropped at the final gate regardless of how cleanly it was detected.

Most investigations that feel mysterious end at step 7. The detection stages are comparatively loud about what they do; the allow-list gate is the one that removes a valid tag without saying a word.