Measure the AI workflow, not how often people open the tool

Imagine a quarterly AI dashboard with these rows:

Signal	Result
Licenses assigned	Increasing
Activated accounts	Increasing
Weekly active users	Increasing
Prompts submitted	Increasing

Now ask the dashboard whether customer replies are more accurate, analysts finish reports with less rework, or managers are spending more time reviewing plausible mistakes.

It cannot answer.

Product activity can show whether people found and opened a tool. It cannot show whether the workflow improved. A rollout can produce excellent adoption charts while moving effort from drafting to correction, creating a new review queue, or quietly lowering the quality of the final work.

The remedy is not a larger AI dashboard. It is a smaller workflow scorecard.

Start before the prompt

Measure the existing workflow before changing it.

For example, suppose a support team wants AI to draft replies. The baseline is not how many agents already use an assistant. It is how the team handles the job today:

• when a ticket enters and leaves the queue
• which cases need escalation
• what reviewers correct before a reply is sent
• what customers ask again because the first answer did not resolve the issue
• which data the team handles and where it is allowed to go
• what the workflow costs to operate, including review and exception handling

The UK Government AI Playbook recommends defining the goal and user need before choosing AI, observing the current process, and establishing baseline metrics against which project outcomes can be assessed. It also draws a useful distinction: model metrics describe how the technology performs, while service metrics show whether user needs and business goals are being met AI Playbook for the UK Government.

That distinction matters for ordinary workplace assistants. A vendor can report latency, uptime, or evaluation scores. Only the workflow owner can say whether a useful case now moves through the business with less delay, fewer defects, and acceptable risk.

If there is no baseline, do not manufacture precision after launch. Use a sampled review of historical work, interviews with the people doing and receiving it, and the operational records already available. Record what cannot be reconstructed. An honest partial baseline is more useful than a confident comparison built from incompatible data.

Give the scorecard two jobs

The scorecard must show value and expose harm. If it does only the first, it becomes sales material. If it does only the second, it becomes a risk register detached from the reason the workflow exists.

A practical scorecard covers these parts of the work:

Part of the workflow	What to inspect	Example signal
Outcome	Whether the user or business result improved	Cases resolved without a second contact
Quality	Whether the final work meets the same or a higher standard	Material corrections found before sending
Human effort	Where time and attention moved, including review, escalation, and exception work	Reviewer time per completed case
Risk and operations	Whether privacy, security, fairness, reliability, and control assumptions still hold	Sensitive-data events, access exceptions, appeals, failed handoffs, downtime

Choose signals that match the decision the team will make. A team deciding whether to renew a drafting assistant needs outcome, quality, effort, risk, and total operating-cost evidence. A team testing whether a narrow task is feasible may need only a small sample of outputs, reviewer notes, and a record of failure modes.

NIST’s AI Risk Management Framework calls for quantitative, qualitative, or mixed methods connected to the deployment context. It recommends documented performance assessment, measures of uncertainty, production monitoring, feedback from users and affected people, and regular reassessment of whether metrics and controls still work NIST AI RMF Core.

This is a better foundation than a universal list of AI KPIs. The right measure for a private research assistant is different from the right measure for a workflow that sends customer messages or changes company records.

Keep usage in its proper place

Usage data is still useful. Treat it as a diagnostic signal.

Low use can point to poor training, missing access, a slow interface, weak output, or a workflow that was never worth changing. High use can point to genuine utility, but it can also reflect a mandate, novelty, or employees shifting work into an approved tool because alternatives were removed.

Ask what usage is supposed to explain:

• Did the intended people try the workflow?
• Did they continue after the novelty period?
• Which task types do they accept, edit, reject, or route around?
• Are a few people producing most of the activity?
• Does greater use correspond with better workflow outcomes, or merely more generated material?

Do not turn prompt volume into a target. People respond to targets. A prompt quota rewards fragmented prompting and penalizes employees who complete the same work with fewer interactions. It also encourages the team to collect more behavioral data than it may need.

Count the work that moved, not only the work that vanished

AI rarely removes a task cleanly. It redistributes it.

A first draft may arrive faster while review takes longer. A summary may reduce reading time while making source verification more important. An agent may complete routine cases while concentrating unusual, difficult cases in the human queue. A customer-facing assistant may answer immediately while creating a new burden for appeals and corrections.

Measure that displaced work. Include configuration, evaluation, access administration, incident response, vendor management, training, monitoring, and fallback operation. Include the time of domain experts who review output, not just the subscription price and the time of the primary user.

This does not require converting every minute into money. A directionally reliable view of who gained work and who inherited it can reveal whether the workflow is sustainable.

It also prevents a common attribution error. If the process changed at the same time as the AI tool, the result belongs to the combined workflow. Do not credit the model for improvements caused by a new template, cleaner source data, fewer approval steps, or better staffing. Those may be the cheaper and more durable intervention.

Measure without building employee surveillance

An AI measurement program can create its own privacy and security problem.

Raw prompts and outputs may contain customer information, internal strategy, employee data, source code, credentials, or material copied from restricted systems. Centralizing that content for analytics expands access and creates another sensitive dataset to retain, secure, and eventually delete.

Start with the least intrusive evidence that can answer the decision:

• aggregate workflow outcomes rather than employee rankings
• sampled output review rather than indefinite storage of every interaction
• task categories rather than raw prompt text
• exception counts with access-controlled case details
• voluntary user research separated from performance management

Document who can see the evidence, how long it is kept, what purpose it serves, and whether the vendor collects an additional copy. If employee-level monitoring is proposed, involve privacy, legal, security, worker representatives, or other relevant functions according to the organization and jurisdiction.

The OECD’s accountability guidance assigns responsibility across the AI lifecycle according to role, context, and ability to act. It treats monitoring, documentation, communication, and consultation as ongoing work rather than a one-time approval Advancing accountability in AI. In practice, the AI program can define the measurement standard, but the workflow owner must interpret the result with the people who perform and receive the work. Security and privacy teams should own their reviews; they should not be reduced to dashboard fields maintained by the program office.

End each review with a decision

A scorecard without a decision rule becomes reporting theater.

Before the pilot or review period begins, write down what would justify expanding, repairing, or stopping the workflow.

Expand when the intended outcome and quality improve, displaced work is acceptable, the control assumptions still hold, and the team can support the workflow at a larger scope.

Repair when there is evidence of value but a specific failure can be addressed: weak source data, unclear instructions, excessive reviewer effort, uneven accessibility, poor escalation, or a control that does not work as designed.

Stop or replace when the outcome does not improve, harms exceed the agreed tolerance, users cannot operate it reliably, the total cost is unjustified, or a simpler non-AI change solves the problem.

NIST includes an explicit management decision about whether an AI system achieves its intended purpose and whether deployment should proceed. It also recommends considering viable non-AI approaches when managing risk NIST AI RMF Core. That is the discipline a rollout dashboard should support.

When this is too much

Do not build a formal measurement program for every private, reversible experiment with public information. A short work diary, a sample of before-and-after outputs, and a team retrospective may be proportionate.

Do not use this scorecard as a substitute for a controlled study when the decision is expensive, high impact, or contested and causal evidence matters. Bring in evaluation specialists and domain reviewers.

Do not force one scorecard across unrelated workflows. Shared categories can make governance comparable, but local owners need measures that reflect their users, failure modes, and operating context.

For mature operational processes, existing quality management, service management, security monitoring, or financial controls may already provide most of the evidence. Add AI-specific measures only where the technology changes the risk or the way work is performed.

The operating rule is simple: license activity tells you whether the product was touched. Workflow evidence tells you whether the change was worth keeping.