Agentic systems need an objective

An agent is not just a tool that produces output. In the canonical definitions, an agent is a temporally persistent system that observes, retains state or memory, chooses actions, and acts in service of an objective.

That objective matters. Without it, the system is only reacting; with it, the system becomes agentic.

This is consistent with the core agent literature: Russell & Norvig treat agents as systems that perceive and act in an environment, Sutton & Barto define the RL agent as the decision-making side of the loop, and control theory treats the analogous object as a controller acting toward a desired outcome.

What goes into an objective

A useful objective should say:

  • what success means
  • what environment or domain it applies to
  • what time horizon matters
  • what constraints cannot be violated
  • what tradeoffs matter when goals conflict
  • what failure looks like

So the objective is not the metric. It is the thing we care about.

For example:

  • “Produce correct, useful, safe software changes over time” is an objective.
  • Test pass rate, bug rate, latency, and review quality are possible measurements or proxies.

Why measurement is needed

If the objective is latent or only partially observable, we cannot usually optimize it directly. We need measurement signals that tell us whether the system is moving in the right direction.

That is why measurement exists:

  • to observe outcomes
  • to estimate hidden state
  • to check whether the objective is improving
  • to detect when the system is gaming the metric

This is the same pattern seen in control theory, reinforcement learning, and measurement theory. In control, the true state is often not directly visible. In RL, reward is an operational signal, not always the full intended goal. In measurement theory, the construct and the measure are not the same thing.

Why proxies are used

A proxy is a measurable stand-in for the objective.

We use proxies because:

  • the objective may be too broad to measure directly
  • the objective may be long-horizon
  • the objective may be hidden inside a larger process
  • the objective may be expensive or slow to verify
  • the objective may require multiple dimensions

So instead of one score, we often need a proxy set.

The working framework

Objective, proxy, measurement, evaluation, and action flow

The diagram makes the separation explicit:

How to write the objective

Write the objective first in plain language. Then separate it from the metric. Then state the tradeoffs. Then state the failure modes.

A good objective includes:

  1. Target outcome
  2. Scope
  3. Time horizon
  4. Constraints
  5. Tradeoffs
  6. Failure modes

How to get proxies from the objective

Start by decomposing the objective into dimensions. Then ask what observable signals should move if the objective improves. Then choose multiple proxies that cover different dimensions. Then define how each proxy is measured. Then validate them against outcomes and failure cases. Then audit for Goodhart risk.

So the flow is:

  1. Define the objective
  2. Decompose it
  3. Find observables
  4. Turn observables into proxies
  5. Measure those proxies
  6. Validate them
  7. Revise when they drift

Where this leads next

The next step is to discuss controls: how to add mechanisms that change state in ways we can measure through proxies. That is where policy, feedback, steering, and stability enter the picture.

Sources

  • Russell & Norvig, Artificial Intelligence: A Modern Approach
  • Sutton & Barto, Reinforcement Learning
  • Åström & Murray, Feedback Systems
  • Cronbach & Meehl, Construct Validity in Psychological Tests
  • Manheim & Garrabrant, Categorizing Variants of Goodhart’s Law

Internal project notes are used to shape this page but are not listed here because the public website does not have access to them.