Grading the Robot Analysts: Opus 4.8 vs Kimi K3 on TTP Extraction

Every CTI team has the same backlog: a pile of vendor reports, CISA advisories, and incident writeups that someone is supposed to read, extract the TTPs from, and map to ATT&CK. It is high-value work and nobody has time for it, which makes it exactly the kind of task people are now throwing LLMs at.

So the question is not "can a model do TTP extraction." It obviously can, sort of. The question is: how accurately, at what cost, and does the expensive frontier model actually beat the cheap open-weight one on this specific job?

I built a harness to find out, and ran it two ways. First on a small hand-labeled warm-up corpus, then on 25 real CISA advisories with the ATT&CK answer key stripped out of the prose. Those two runs told noticeably different stories, and the difference is the most useful thing in this whole exercise: on the easy set the models bunched together and the cheapest one came out on top, and on the hard, realistic set a clear and stable quality ladder appeared. The harness and both corpora are on GitHub if you want to check the numbers or run your own.

The contenders

Four models, two closed and two open-weight:

Two notes before the numbers. K3 and DeepSeek run through Fireworks' OpenAI-compatible endpoint, so their prices here are Fireworks' hosted rates. And the structured-output row is not cosmetic: Claude can be schema-constrained so malformed output is impossible by construction, while the others are asked for JSON and validated after the fact. That difference is invisible on easy inputs and very visible on hard ones, as we will see.

Why TTP extraction is a good benchmark

Three reasons this task is a better model eval than another coding leaderboard:

• There is real ground truth. A technique either is or is not described in the text. You score precision and recall against a labeled mapping instead of asking another model to judge vibes.
• It punishes both failure modes. Hallucinated techniques destroy analyst trust (precision), and missed techniques defeat the purpose (recall). A model has to be good at both to be useful.
• It is a real workload with a real budget. Nobody runs one report. The unit economics of ten thousand reports a year are part of the answer.

Two corpora, on purpose

The warm-up: 20 hand-labeled reports. A seed set of short reports, 121 gold labels across roughly 85 techniques, labeled by hand to the behavior the text explicitly supports. Useful for shaking out the harness, but, it turns out, too easy to separate good models from great ones.

The real test: 25 CISA advisories. This is where the interesting numbers come from. CISA's AA-series advisories ship a per-tactic MITRE ATT&CK technique table, which is document-level, multi-label ground truth for free. The ingestion script fetches each advisory, reads the gold technique ids out of those tables, and then strips every table out of the prose, so the model cannot read the answer key and has to infer techniques from described behavior alone. That yields 635 gold labels across 25 advisories, from 3 to 67 techniques each (mean 25), spanning 257 unique techniques. This is the production task exactly: real advisory prose in, ATT&CK ids out.

The task and scoring are identical across both. Every model gets the same system prompt and instructions, verbatim, and must return a list of technique ids each with a short quoted evidence span. Scoring uses two F1 numbers: strict, which demands an exact id match, and parent, which collapses sub-techniques to their parent so "right technique, wrong sub-technique" still counts. Precision and recall are micro-averaged. Refusals and malformed responses are tracked separately. The two corpora are scored as separate splits and their numbers are not comparable to each other: different difficulty, different labeling philosophy.

Results: the real advisories

Run on the 25-report CISA corpus, 2026-07-29:

Every model roughly halved its F1 against the real advisories compared to the warm-up, and that is exactly the point. The tight bunching of the easy set opens into a clear, stable ranking: Claude, then K3, then DeepSeek, then Qwen3-Max. The order does not change even if you throw out the reports where a model produced invalid JSON (0.535 / 0.505 / 0.441 / 0.372). Reading the columns:

Claude Opus 4.8 is the quality leader, at roughly 2.7 times K3's cost and 12 times Qwen's. Notably, it refused nothing here across all 25 advisories, and I will come back to that, because it refused a report on the warm-up.

Kimi K3 is the value pick. It lands within about 0.03 F1 of Claude at a third of the cost, with zero refusals and reliable JSON on every advisory. For bulk ingestion, "97% of the quality for 37% of the price, no reliability surprises" is the profile you want.

Qwen3-Max finished last, despite being Alibaba's proprietary flagship. Worst recall in the field at 0.290, meaning it finds under a third of the gold techniques, and it produced invalid JSON on one of the densest advisories. It is the cheapest per report, but on quality-per-dollar it is dominated by K3, which costs about 4.5 times as much for 0.14 more F1 and zero failures. The open-weight model beat the closed flagship outright.

DeepSeek V4 Pro is hard to recommend either. Low recall (0.335) and, more damning, invalid JSON on the two most technique-dense advisories. That is a reliability cliff on long, complex inputs, not simple truncation, and it is the failure mode that matters most in an ingestion pipeline. Both budget models crack on the hardest real inputs while Claude and K3 stay clean across all 25.

Everyone loses 0.12 to 0.15 F1 to sub-technique granularity. The parent-F1 column sits well above strict for all four. If your downstream graph only needs technique-level resolution, every model is meaningfully better than the strict numbers suggest, and the cheaper ones close some of the gap.

By threat category

Slicing the 25 advisories by type is where the model choice gets a useful sanity check. Strict F1, with recall in parentheses:

Three things fall out of this, and the first one matters most for how you deploy:

• The ranking holds in every category. Claude beats K3 everywhere. There is no report type where a cheaper model wins, which means there is no routing arbitrage: you cannot save money by sending ransomware reports to one model and APT reports to another. Backend model choice is a single decision, not a per-report-type one.
• Difficulty is driven by report type, not just by model. Every model scores far higher on ransomware (around 0.6) than on APT and nation-state advisories (around 0.45). Ransomware advisories follow CISA's standardized #StopRansomware format with canonical, well-trodden TTPs; APT advisories are bespoke technique sets wrapped in freeform narrative. So a corpus's category mix drives its absolute F1 as much as the models do. Read any single benchmark number in light of what it is made of.
• The frontier premium buys the most on the hardest material. K3 is essentially level with Claude on structured ransomware (0.61 versus 0.63), and Claude's edge widens on hard APT. If your intake skews toward messy, novel, nation-state reporting, the quality gap is where you will feel it. DeepSeek makes the same point in reverse: its collapse is concentrated in APT (0.24 F1, 0.18 recall, and both of its JSON failures landed there), so it is least reliable exactly where the hard intelligence lives.

The per-category counts are small (8, 9, 4, and 4), so treat ransomware and APT as the reliable slices and the two smaller categories as suggestive.

What the easy corpus got wrong

Here is why the two-corpus design earned its keep. On the 20-report warm-up, the same four-ish models bunched between 0.84 and 0.93 F1, and Kimi K3 came out on top (0.927) while Claude (0.906) came second, dragged there by a single refusal that zeroed one report's seven gold techniques. Read on its own, that run says "the cheap open model wins." The CISA run says the opposite about quality: on inputs hard enough to separate the field, Claude leads and the ordering is stable.

The lesson is not subtle. A benchmark that is too easy bunches every model near the ceiling, where noise, a lone refusal, a couple of unlucky sub-technique misses, decides the ranking. You cannot tell a 0.906 from a 0.927 model on twenty easy reports; you can tell a 0.53 from a 0.36 model on twenty-five hard ones. If you are choosing a model off a leaderboard where everyone scores in the 0.9s, you are choosing on noise.

The refusal, reconsidered

On the warm-up, Opus 4.8 returned stop_reason: refusal on one benign synthetic ransomware vignette, which in an ingestion pipeline is silent data loss: seven techniques that were in the report never reached the graph, with nothing upstream flagging the gap. That looked like a real strike against deploying Claude for bulk ingestion.

The CISA run reframes it. Across 25 real advisories, some describing extremely nasty state-sponsored intrusions in detail, Claude refused nothing. The refusal appears tied to the compact, synthetic, attack-recipe framing of the seed report, not to real threat intelligence, which comes wrapped in defensive context and attribution. That is a genuinely useful thing to know, and it is the kind of correction you only get by running more than one corpus: the refusal risk is real, but it is triggered by a shape of input your production pipeline is unlikely to see.

The reasoning dial goes the wrong way

One finding held across the board and is worth stating plainly: turning reasoning up hurts this task. On the warm-up, raising K3's reasoning effort to high dropped its F1 to 0.870 and its precision to 0.833, for roughly three times the cost, because the extra reasoning talks the model out of correct parent techniques and into wrong sub-techniques. Extraction is a grounding task, not a reasoning task: you want the model reading the text, not theorizing about it. That is why every model here was run reasoning-minimal. If you are paying for thinking tokens on this workload, you are paying to make the answer worse.

Cost

Real advisories are long, so per-report costs are much higher than the toy set, and the gaps matter at scale:

TTP extraction is the canonical batch workload, thousands of independent documents with nobody waiting on any single response, so the fair comparison gives Claude its 50% Message Batches discount. Even then, batched Opus ($460 per ten thousand advisories) stays about a third above K3 ($340), and K3 is only 0.03 F1 behind. So the honest framing is: Claude buys you the top of the quality ladder at a real and persistent premium, and K3 buys you nearly all of that quality with better cost and no reliability asterisks. The budget models are cheaper still, but you are paying in recall and in JSON that fails exactly when the advisory is hardest.

What I would actually deploy

• If quality is the only axis that matters, Claude Opus 4.8. It leads on the hard corpus, stays clean on every input, and its one refusal mode is triggered by synthetic framing you are unlikely to feed it in production. You pay a persistent premium for the top of the ladder.
• For almost everyone, Kimi K3 is the default. Within 0.03 F1 of the leader at a third of the cost, zero refusals, reliable JSON on all 25 advisories including the 67-technique monster. It is the quality-per-dollar sweet spot, and being open-weight it can be self-hosted, which for shops handling TLP:AMBER reporting is a governance decision that outranks any F1 delta.
• Be wary of the budget models on dense inputs. DeepSeek and Qwen3-Max both produced invalid JSON on the most technique-heavy advisories, and both under-extract. They are fine for short, simple reports and a liability on the long, complex ones that carry the most techniques.
• Tier by priority or confidence, not by report type. Claude wins every threat category, so there is no report-type routing that saves money. But a cheap reliable model on the first pass, with a frontier model reserved for high-priority actors or low-confidence extractions, still captures most of the accuracy for a fraction of the cost.
• Do not pay for reasoning on this task. It is a cost multiplier that lowers quality.

Caveats

• CISA tables are not exhaustive. Precision sits near 0.50 for all four models partly because they extract real techniques that are described in the prose but were not in the curated table, which scores as a false positive. That deflates precision roughly equally across models, so trust the ranking more than the absolute F1.
• The two corpora are not comparable. Different difficulty, different labeling philosophy. Score them as separate splits, which is how they are reported here.
• Absolute F1 depends on the corpus category mix. Ransomware advisories are far easier than APT ones, so a corpus weighted toward either shifts every model's headline number. Compare models within a fixed mix, not across corpora with different compositions.
• Both corpora are small. Twenty and twenty-five reports are enough to be directional, not definitive. Do not bet a budget on a 0.03 F1 gap without expanding them.
• Contamination is possible. Real advisories may sit in a model's training data. Stripping the ATT&CK tables from the prose mitigates it, the model cannot copy the answer, but a model that has seen the campaign write-up before still has an edge.
• Models and prices move fast. The CISA run is dated 2026-07-29. Re-run the harness before relying on any of it.

Harness

The harness, both corpora, the CISA ingestion script, the shared prompt, the schema, and the scoring code are all in the ttp-benchmark directory of AEGIS Labs. Adding a model is one entry in config.yaml. If you want to argue with these numbers, that is the right way to do it: expand the corpus, re-run every model, and see what holds.

The Attack Surface covers threat intelligence, offensive security, and AI security.