Building the Systems That Produce Software

In my current workflow, coding agents follow problems across repositories, deployments, data and external services. They inspect runtime behavior and return to the implementation with a tested hypothesis.

Following a problem this far turns implementation into one step of a broader loop. The outcome also depends on the context the agent receives, the systems it can access, the feedback it can act on and the engineer’s judgment.

Engineers will need to build systems that allow humans and agents to produce software reliably.

Start with outcomes and constraints #

Engineering tickets are often written as implementation tasks: add an endpoint, introduce a cache or put another button on a page. This works when the approach has already been evaluated or the task is deliberately scoped for a junior or mid-level engineer. When the solution remains open, the ticket should instead state the desired outcome and constraints. Any proposed implementation becomes a hypothesis to validate.

Consider a query that slows as its dataset grows. “Add an index” prescribes one solution. “Return results within 500 ms for representative production-scale data without significantly slowing writes” defines the actual problem and its acceptance criteria.

This framing gives engineers and agents room to evaluate options such as indexing, query design, schema changes or pre-computation while providing concrete criteria against which each option can be tested.

Product and engineering need to define the problem together. Product brings customer behavior and business constraints. Engineering makes failure modes, operational limits and technical trade-offs explicit. Once the outcome and constraints are clear, the agent can explore implementation options without silently inventing the goal.

Build a complete feedback loop #

The useful unit of AI-assisted work is shifting from a generated diff to a complete problem-solving loop:

Orange marks steps that require human judgment. Blue marks work performed by the agent or verification system.

A team’s existing development workflow provides baseline verification through tests, static analysis and deployment checks. For each new change, the engineer defines what additional evidence would make the result trustworthy and extends the existing checks where needed. The agent investigates and implements against those criteria. The engineer then interprets the results and improves the surrounding system with what the work revealed.

CAVE: four capabilities for agentic engineering #

These capabilities do not come from the agent alone. Teams must provide the surrounding tools, access controls and workflows.

I group these requirements into four capabilities:

• context to know the system,
• access to reach relevant systems and signals safely,
• verification to check outcomes and
• engineering memory to retain validated knowledge.

1. Context #

Agents need to know how services relate, which patterns are intentional, where architectural boundaries sit and which operational constraints matter. Repository instructions, architecture decisions, ownership information, security rules, schema documentation and runbooks can provide this context during work.

In my workflow, one example is a searchable knowledge base containing processed meeting transcripts and relevant documents. During ingestion, metadata such as dates, participants and topics is extracted into structured fields. Filler and redundant transcript syntax are removed to make retrieval more precise and token-efficient. Agents can then retrieve product decisions, constraints and prior reasoning without copying the context into each prompt. The repository’s AGENTS.md points agents to this knowledge base, GitLab work items and other approved sources, giving them a stable entry point for finding relevant context.

Context must remain current, scoped and accessible. Stale documentation misleads humans and agents, while a concise repository instruction can improve every future change. Repeated wrong assumptions may justify a new instruction or architecture check; one unusual task may not. Engineers decide which lessons deserve to become shared context.

2. Access #

Many failures only become visible outside a local test run. Deployed code may differ from the branch under investigation, behavior may change as a service scales across replicas and real data or third-party limits may invalidate local assumptions.

In my current workflow, agents can inspect deployment state, read service logs and query development data through approved tools. They can trigger a request themselves or observe one that I initiate through API calls or the UI, follow it through several services and compare the observed behavior with the implementation.

This access has strict boundaries. Agents can read development and test systems, but production credentials are not made available to them. They are instructed to highlight deployment changes and never push by themselves. Destructive operations require explicit approval, even on development systems.

3. Verification #

Agents need fast and relevant feedback. Existing unit tests, type checks and static analysis provide a general baseline. Tell agents how to run these checks or expose them through a Makefile target. For each problem, the engineer identifies important assumptions those checks do not cover and adds targeted verification.

Targeted verification can be lightweight: a curl script that exercises an endpoint, a load script combined with memory and CPU profiling or a repeatable concurrency test that runs overlapping operations. Custom shell scripts serve a different purpose by flagging repository antipatterns or enforcing relevant code-quality thresholds. These static signals constrain an implementation, but do not prove its runtime behavior.

Useful checks become reusable. Replaying a previously failing workflow can prevent the same regression, while an architecture check can prevent invalid dependencies in future changes.

These checks turn vague goals into observable targets. Instead of asking an agent to “make this endpoint faster,” the engineer can define representative requests, expected results, a load profile and latency or memory limits. The agent can iterate against those results, while the engineer remains responsible for whether the scenario and metrics represent the real problem. A passing test or improved benchmark provides useful evidence only when it measures the right behavior.

4. Engineering memory #

Engineering memory is validated knowledge kept in a form that humans and agents can use. Work on a feature may produce a regression test for a bug, a fixture that reproduces tricky data, an operational check, an update to AGENTS.md or a short architecture decision. Engineers do not need to preserve the whole conversation. They summarize and keep the result that will help with the next change, then update or remove it when the system changes. Future agents can start with what the team has already learned. This also helps the team understand the behavior and decisions that led to the current state. Flowcharts, decision trees and other visualizations are really helpful for this.

Use measurement where uncertainty justifies it #

Agents make it cheaper to explore alternatives. For a performance-sensitive query, an engineer can ask an agent to implement two plausible options and benchmark both with representative data.

Multiple implementations still cost time and attention. Many tasks have an obvious solution that fits established patterns. Experiments provide the most value when uncertainty is meaningful and the decision can be evaluated against relevant criteria.

Measurement informs the decision without replacing judgment. Latency, memory use and infrastructure cost can be measured directly. Maintainability, reversibility, consistency and product fit still require an engineer to interpret the evidence and make the final choice.

Start with one recurring workflow #

Building this system does not require a large internal platform project. A team could start with the slow database query from the earlier example:

1. Capture a baseline with a representative dataset in a development environment. Use a curl request to verify the endpoint response, a load script to record latency, memory use and query count, and the database tools to capture the execution plan.
2. Define success before changing code: keep p95 latency below 500 ms without reducing write throughput by more than 10% from the baseline.
3. Expose the baseline check through the repository’s standard verification entry point so the agent can discover and run it.
4. Let the agent implement one or more relevant options, such as adding an index or rewriting the query, then evaluate each against the same baseline script.
5. Choose based on the evidence and record why in a mini ADR, including the alternatives considered and measurements that informed the decision.
6. Keep the selected benchmark and regression test so later changes can be checked against the same behavior. Add reusable constraints to the relevant repository instructions or runbook, and update or remove anything that became outdated. Mark replaced ADRs as superseded rather than deleting them.

This creates one bounded feedback loop around a real problem. The team can apply the same approach to another recurring workflow once the first loop proves useful.

What changes for engineering teams #

As agents take on implementation, engineers remain responsible for decisions the agent cannot make from code alone: which outcome matters, which constraints are real, what evidence is sufficient and which lessons should become part of the system.

The output of each change is more than code. A benchmark, regression test, repository instruction or decision record can make the next change easier to execute and verify. Building these assets task by task turns individual agent runs into a reliable engineering workflow.