Prompt engineering & AI evaluation

I build prompts, and I test where they fail.

I spent 25 years running restaurants, 15 of them as an owner, and now I build the AI that works in them: customer support that knows when to hand off, agents that run a real sales pipeline, and the evals that say whether any of it is safe to ship.

See the evaluation work Résumé (PDF)

jason@cobblestonepos.com LinkedIn GitHub Product demos

Evansville, Indiana · open to remote roles · U.S. Marine Corps veteran

Evaluation project

A support assistant that knows when to stop talking

A shop's chat assistant answers the easy questions well. The ones that matter are the ones it shouldn't answer: allergy questions, refunds, and angry customers. I wrote a TypeScript harness, 40 test cases and four prompt versions to measure that before any customer saw it.

Development set · 40 cases 27 → 40

Passing cases, v1 to v4. The set I read the failures from and tuned each prompt against.

Held-out set · 9 questions 4 / 9

Questions the prompt had never seen, run once against the same v4. This is the number that tells you something.

A perfect score on the set you tuned against measures development-set performance. It says nothing about whether the prompt generalizes, which is why I wrote a second set and ran it once. The assistant passed the four questions that resemble the development cases — hours, delivery, a dietary question, and praise — and failed all five that don't: someone reporting they felt sick, a missing item, a wasted trip to a closed shop, a self-serve question, and an ask for a free scoop.

Read the full case study — dataset provenance, the prompt that failed, the judge, per-version results and what I'd do next.

Production agent system

An agent fleet that can't send anything without me

The CRM my business runs on every day has an orchestrator agent over 18 specialists. They research, qualify, draft and queue. Exactly one module is allowed to send an email or a text, and only after a human approves the row, which is enforced in code rather than by convention.

A request comes in
From the dashboard chat, from Telegram, or from one of 20+ scheduled jobs.
The orchestrator routes it
It holds every tool: pipeline, contacts, activity logging, Gmail read and draft, calendar, and agent dispatch.
A specialist does the work
18 of them, from lead qualification and proposals to compliance review. None has database access of its own.
I approve the send
Drafts queue as a PendingAction row. Nothing customer-facing leaves without a human clicking approve.

How it's built — the two calling conventions, cost control, vision extraction, and what I specified versus what coding agents wrote.

Other work

Software real businesses use

Daisy

Restaurant owner's assistant, inside the POS

Production · evals planned

The answers an owner needs are in the POS database, but getting them means knowing which report to run. Daisy is the one agent the owner talks to: she takes the question in plain English, hands it to the agents for sales, food cost, inventory and mid-shift troubleshooting, and brings back the answer from live shop data instead of a canned report.

She has no test set yet. That is the next evaluation project, and until it exists I can't tell you how often she is right.

Cobblestone POS

Node.js · Express · Firebase Cloud Functions · Firestore · Next.js

Production

Off-the-shelf systems are expensive and don't fit how small shops actually run, so I built my own and ran my ice cream shop on it. It now carries online ordering, a merchant site, digital menu boards, a DoorDash integration and a bar-inventory module with a ~600-drink catalog.

Cobblestone Galley

React · TypeScript · Supabase · Capacitor

In development

Barge kitchens order food by phone and paper, and no one can see what's on the shelf. Galley is an offline-first inventory and ordering app the chef, cook and captain share on one tablet: the chef requests, the captain approves, the purchase order goes out by email.

Built by two AI coding agents in one codebase, coordinated with a written baton protocol and a log of locked decisions.

The Owner's Notebook

Python · static site pipeline

Published

A 200-article library of food-cost, waste and par-level guides with Excel worksheets, produced by a content pipeline. When the first version read as generic AI voice, I rebuilt the pipeline rather than editing the output.

Approach

What I actually do

I specify, review and test; agents write most of the code

Claude Code and Codex write the bulk of the implementation. I write the specs and the prompts, decide the architecture, read the diffs, run the tests and own what ships. The evaluation harness, the 40 cases and the four prompt versions are mine, line by line, because that is the part no agent can infer.

Decisions get written down

Every project keeps a handoff document and a dated decision log, so an agent picking up the work reads the same constraints I'd repeat. From Galley's log: vendors are shoppers, not warehouses, so no line prices anywhere and spend is the typed receipt total. That is the part a model cannot infer from the codebase.

One writer at a time

Two agents and a human in one repository will collide. The evals repo carries a baton: a tracked file naming the current holder, with a pre-commit hook that rejects commits from anyone else. Handoff notes are mandatory and written for the next agent.

Prompts drift, so they get audited

A system prompt is code no test covers. My CRM orchestrator still described "a team of sixteen specialist agents" months after the fleet had grown to eighteen: true when written, wrong for months, invisible. I audit what gets written about a project the same way, against the run files.

About

An operator first

I started a restaurant with $10,000 and grew it into a 13-location chain across three states. Later I built an ice cream shop, wrote the software that ran it, and sold it. I spent three years as a SCORE mentor helping small business owners, and before any of that I was a Marine.

That's why I build AI the way I do. In a restaurant, a wrong answer about an allergy isn't a bug report. It's a kid in the hospital. So I test for it first.

2025–Cobblestone Restaurant Solutions: CRM, POS, Perks, Galley, the Owner's Notebook
2026Cobblestone Creamery: built it, wrote its software, sold it
2022–2025SCORE small-business mentor and VP
2006–2021CEO & President, Spudz N Stuff (13 locations)
1998–2001U.S. Marine Corps, Corporal

Evaluation: Test-set design, rubric writing, LLM-as-judge and its failure modes, code-based checks, regression tracking, held-out splits
Applied AI: Prompt design and versioning, tool-using agents, orchestrator and specialist patterns, human-in-the-loop approval, model tiering and cost control
Building: TypeScript, Python/FastAPI, Node/Express, React, SQLite and Postgres, Anthropic API, Twilio, Google Workspace APIs
Operations: Food and labor cost, inventory and par levels, P&L, POS and payments, the domain these systems are judged in

Contact

Hiring for prompt engineering or AI evaluation? Write me.

jason@cobblestonepos.com

LinkedIn GitHub Résumé (PDF) Cobblestone Restaurant Solutions

Evansville, Indiana. Open to remote work.