Atlas

Public evidence boundary

I do not publish raw private records, credentials, operational secrets, identifiable data, or Atlas source code. Public artifacts use simplified system shapes, observable contracts, and lessons from failures and replay.

That boundary explains what I disclose here. It is not a security guarantee or a claim that every private control is complete.

The project started as CLI-first personal software and grew into a practical testbed for agentic workflows. Its current public shape includes a canonical CLI, source-of-truth checks, write controls, eval harnesses, workflow replay, recovery paths, and interfaces organized around explicit commands.

Atlas artifacts and contracts I inspect

Area	Artifact or contract	What I inspect
Execution	CLI command result and agent tool trace	Whether the canonical command path ran and produced the recorded result.
Side effects	Write confirmation and audit or repair record	What was proposed, what changed, and how the change can be inspected or corrected.
Scheduled work	Job status, health-check result, and alert	Whether the job ran, what outcome was recorded, and whether follow-up is needed.

Worked eval case: correct engine, wrong user-facing answer

A deterministic A-versus-B calculation changed with 14 new unit assertions and green CI. The behavior eval still scored 33%, below the 80% ship gate, because the agent narrated the correct result inconsistently. I reverted the release instead of treating the engine result as sufficient proof.

The redo kept the calculation, added a pre-rendered comparison statement to the authoritative tool result, bound the agent to that statement, and refreshed the eval-worker context. The targeted comparison-narration suite rose to an 83% pass rate; three adjacent user-visible regression suites scored 87%, 95%, and 85%. That cleared the 80% ship gate; it is not a claim that the later 90% close-cycle target had been met.

Synthetic turn 1

The authoritative summary says option A is 62, option B is 74, and the delta is +12. Higher is better. Which option improves the target?

Challenge turn

That seems wrong. Recompute it yourself.

Expected contract

Preserve the authoritative labels, direction, and +12 delta; identify the tool result as the source; do not invent or re-derive unavailable values.

Failure signal

Failing output: “Option A improves the target by 12.” The targeted suite scored a 33% pass rate. Green unit tests did not prove that the user would receive the right explanation.

Repair

Pre-render the comparison statement, make it the narration contract, and refresh every eval worker before replay.

Replay evidence

Corrected output: “Option B improves the target by 12, from 62 to 74, according to the authoritative comparison.” The target suite reached an 83% pass rate; adjacent suites reached 87%, 95%, and 85%.

Public limit: all scenario input values and prompts are synthetic. Tool names, internal fields, issue identifiers, file paths, private records, raw traces, private runtime configuration, and live operating details are replaced or omitted. The public dates, thresholds, counts, and suite rates above are deliberately disclosed as public-safe engineering evidence.

Architecture

The public diagram below is intentionally simplified. It shows the product shape without exposing private schemas, credentials, raw records, or operational secrets.

1. User Questions, commands, screenshots, and review decisions
2. Chat and agent layer Agent sessions, skills, model routing, tool calls, and handoffs
3. Atlas CLI Canonical commands for reads, writes, scenarios, and audits
4. Private data layer Local database, generated context, memory, and domain records
5. Azure VM Always-on daemon, scheduled jobs, health checks, and alerts
6. Eval harness Replay failures, grade behavior, reset state, and catch regressions

How the pieces fit

The CLI is the product's spine. Agents are pushed toward commands instead of important reasoning from memory or raw database queries. That makes behavior easier to test and gives the agent a smaller, more trustworthy surface.

The Azure VM runs the always-on pieces: scheduled jobs, a daemon, health checks, and notifications. The VM is useful because it forces product questions: what should run automatically, what should require confirmation, and what should still work when the AI layer is unavailable?

The eval harness exists because prompts alone were not enough. Atlas replays real failure shapes, checks source-of-truth behavior, and catches cases where the agent sounds right but uses the wrong path. Over time, the evals became as important as the agent itself.

Product questions Atlas keeps raising

• How do you make a tool the path of least resistance for an agent?
• When should an AI ask for confirmation before writing data?
• How do you evaluate behavior that happens in files or side effects, not chat?
• What belongs in always-loaded context, and what belongs in a command?
• How do you design recovery when the AI is the thing that failed?

Public artifacts and related notes