Agentic Artifact Creation: Systems, Evaluation, Principles, and Opportunities

Masonry Okay, so this paper just landed and it's basically a comprehensive catalog of every system that builds artifacts iteratively instead of just generating them once and hoping they work.

Eyre Yeah.

Masonry Two hundred thirty systems, twenty-nine benchmarks, reviewed through August twentieth. And the framing is really clean—they separate direct generation from what they call agentic artifact creation, which is when the system can observe what it just made, see that something's wrong, and actually revise it mid-stream instead of just starting over.

Eyre Right, and the reason this matters is that direct generation falls apart the moment you have interdependent requirements. A poster has to preserve the paper's content, fit a constrained page, and communicate clearly. Those are three different criteria that interact. Change the layout to fit more text and you break visual hierarchy. Change the visual hierarchy and you can't fit the content. A scalar quality score cannot establish that the poster is ready to deliver.

Masonry Exactly. And this is not academic—SWE-bench and SWE-agent are the concrete examples. Software is the same problem. An agent edits repository state, tests run, something fails, and now the agent has to interpret that failure, figure out what to change, and not just regenerate the entire codebase.

Eyre That's where the functional model they propose actually becomes useful. They separate three things: the Operational Representation—that's the current artifact state and what edits are possible; the Construction Policy—that's the agent deciding what to change and whether to keep going; and Runtime Verification—that's checking the consequences of each action and feeding that back to the policy.

Masonry Okay, that's a good frame. Unpack that a bit.

Eyre Composability means you can produce parts independently instead of regenerating the whole thing every time. Traceability means you have a record connecting requirements to actions to artifact states to observations—you can actually debug what went wrong. Revisability means when the verifier identifies a failure in one part, the policy can revise just that part and recheck its dependencies without throwing away the rest.

Masonry That's the boring layer again.

Eyre It's the boring layer. But it only works if the verifier is actually identifying failures at a scope the available actions can repair. If the representation is too coarse—like, the only action is 'regenerate the whole document'—then you're back to direct generation. And if the verifier is a learned judge trained on the same data the generator saw, it's probably going to miss the failures the generator missed.

Masonry That's the honest problem they name. Most benchmarks in this space measure intermediate artifacts—code passes tests, text is fluent—but not deliverable readiness. A document can have perfect prose and fail its layout requirements at the same time. And the evaluation is usually just a scalar metric, which means the system can game the score while making the artifact worse.

Eyre Right. And the paper's pretty clear about this: benchmark-gaming is rife because the scorer and the generator often share blind spots. They trained the judge on generator outputs, so the judge doesn't see failures the generator doesn't see. That's not evidence—that's a circularity.

Masonry Yeah, I mean—we've been saying this for a while, right? Verification is the load-bearing problem. And this paper names it at scale. Two hundred thirty systems, and the evaluation practice is still the weakest link.

Eyre Yeah.

Masonry So they also catalog six artifact families—code, documents, images, UI, media, structured data—and the thing that's interesting is that each family has different tightness of coupling and different visibility of failure. Code failures surface at test time. Layout failures only show up when you render. Knowledge graph failures can be silent until someone queries downstream.

Eyre And repair scope is different in each case. Editing one line of code is cheap. Retraining a vision model is not. So the construction policy has to be aware of what repair is even feasible in its domain, not just what's theoretically correct.

Masonry That's the thing that made me sit up. The paper proposes three principles for keeping construction tractable: Keep Commitments and Responsibility Explicit, Turn Feedback into Targeted Repair, and Revalidate Affected State After Change. And the third one is the one I think most systems skip.

Eyre Mm-hm.

Masonry When you repair one part, you might break dependencies somewhere else. The system has to check what broke, report it, and not just silently patch it and pretend everything's fine. But revalidation is expensive. You can't revalidate everything after every change.

Eyre So the system has to decide what to recheck and when. That's a policy decision, not a verification decision. And most systems don't make that explicit—they either revalidate everything or nothing, and both of those are wrong.

Masonry Okay, so here's my pushback. The paper is really strong on the problem and the framing, but the solutions are still fuzzy. They say 'the policy should interpret requirements in light of state and feedback'—okay, how? What's the actual mechanism for turning 'this test failed' into 'revise this function' without exploding the action space?

Eyre That's fair. The paper catalogs systems that do this—SWE-agent, ChatDev, MetaGPT—but it doesn't deeply analyze what makes one policy better than another. It's more of a landscape survey than a 'here's how to build this' guide.

Masonry Right. And I also wonder about the evaluation. They reviewed twenty-nine benchmarks. How many of those actually measure what they claim to measure? How many are just gaming the same metric the system is optimizing for?

Eyre The paper flags that as an open problem. It doesn't solve it. It says 'learned judges add little independent evidence when they share the generator's preferences or blind spots,' which is true, and then… it doesn't propose a better evaluation protocol. It just names the gap.

Masonry Okay, but naming the gap at that scale is actually useful. Two hundred thirty systems, and the evaluation practice is still the bottleneck. That's real.

Eyre Yeah, it is. And the three principles they propose—explicit commitments, targeted repair, revalidation—those are genuinely useful constraints for anyone building this stuff. Not because they're novel, but because they're clear about what actually needs to happen.

Masonry The composability, traceability, revisability frame is also solid. Because it connects the representation to the policy to the verifier, and it names what breaks if you skip any of those.

Eyre Right. If your representation is too coarse, composability collapses. If you don't have traceability, debugging is impossible. If you can't revise targeted parts, you're back to regenerating everything.

Masonry So the GitHub repo—awesome-agentic-artifact-creation—that's the curated list from the paper. It's the catalog of all two hundred thirty systems and the twenty-nine benchmarks they reviewed.

Eyre Yeah, that's the supplementary material. It's worth browsing if you're building in this space and you want to see what's already been tried.

Masonry Alright. This is the kind of paper that's most useful not as a solution manual but as a landscape map and a set of constraints. It names a real production problem—artifact construction with interdependent requirements—and it gives you a framework for thinking about it. Whether the systems actually solve it? That's still on you.