Case study — tiehack works intelligence platform

Multi-Agent Operations Platform

An entire marketing department, on call. I tell a director what I need in plain language; it dispatches the right specialist agents, each pulls real data with its own tools, and their work is quality-checked before it comes back to me. The platform that runs my agency: ~20 domain agents behind one conversation.

~20

domain agents

115

registered tools

services

~50K

lines

in practice

What it's actually like to use

I open a chat with the director and talk to it the way I'd brief a team lead: "I've got a new brand — here's the URL and the social handles. Run me a full technical SEO and social audit." It decides which specialists the job needs, hands the work to each, and they go execute. The audit agent runs Lighthouse and pulls domain data, the SEO agent checks rankings and keywords, and the social agent scrapes the handles' public metrics via Apify (followers, engagement, recent posts).

Every piece comes back through the director, which runs it past a quality judge before anything reaches me, and sends work back to be redone if it doesn't clear the bar. What I get is one reviewed answer. What it replaces is a department I'd otherwise have to staff, brief, and chase.

The hard problem with agents over business data isn't generating text. It's making sure the text is true, and good enough to put in front of a client. The whole architecture is built around those two constraints.

The practical outcome: one person runs a multi-client agency with a department's worth of specialist coverage on call.

architecture

How it's put together

how a request flows

you →

"Run a full technical SEO and social audit for this brand — here's the URL and the handles."

director

Reads the brief, picks the specialists it needs, and hands off to each.

SEOTechnical auditSocialBrandPaid searchResearch

Each specialist runs its own tools against live data — it can't answer before it has pulled real numbers.

PostgresGoogle AdsGA4SEO / SERPApifyShopifyLighthouseweb search

quality gate

The director assembles the pieces; a judge scores each 0–100. Anything below bar is sent back to revise ↺ before you ever see it.

→ you

One reviewed report back — the whole department, on call.

▪ every tool call → Postgres audit log▪ daily API-cost kill switch

The director: a meta-agent you talk to

Built on Anthropic's Agent SDK. It interprets a plain-language brief, decides which specialists a task needs, and hands off to them via native sub-agent handoff, then assembles what comes back.

The specialist fleet: ~20 domain agents

One agent per domain (paid search, SEO, social, brand, audit, content, research, email, and more). Each is a domain system prompt plus its own MCP server exposing a scoped set of tools. The director invokes only the specialists a job actually calls for.

Grounded execution

Each specialist runs a gather → respond pipeline with enforced tool use: it must pull live data before it can answer. 115 tools across the fleet (Google Ads, GA4, SERP, Shopify, Lighthouse, web search, the knowledge base), each scoped to the right agent.

Quality gate

An LLM-as-judge scores every piece 0–100 on a 7-criteria rubric and returns a verdict; work below the bar is sent back to revise. Scores persist, so I can track each agent's quality trend over time.

Reach it from anywhere

Talk to the fleet from team chat (self-hosted Matrix), an HTTP API, or a custom stdio MCP server that exposes it to my coding environment (list_agents / ask_agent).

Observability & safety

Every tool call is logged to Postgres for a full audit trail; a daily API-cost kill switch halts the fleet if spend crosses budget; Sentry and healthchecks.io watch all services.

the roster

Twenty specialists, organized like a department

The fleet isn't twenty copies of the same assistant. It's a staffed org. Four agents each own a client account; the rest are shared specialists grouped by what they actually do. The director pulls whoever the job needs.

Client account managers

Each is embedded in one account and trained on that client's full context — platforms, business model, growth model, the P&L, content brief, and brand voice. They answer as someone who knows the account.

›Events venue — paid + bookings + events pipeline
›DTC lifestyle brand — Shopify, content, keywords
›Coworking operator — leads, blog, competitive intel
›The agency itself — content, newsletter, thought leadership

Research & competitive intelligence

Track the market and competitors so recommendations are grounded in what rivals are actually doing.

›Research — web + knowledge base; tracks 100+ DTC brands
›Brand intel — competitor behavior via email monitoring
›Email intel — competitive teardowns across 200+ brands
›Meta ad intel — competitor Meta/Facebook creative analysis

Search, technical & visibility

Diagnose organic, technical, and AI-search health for clients and prospects.

›SEO — rankings, keywords, backlinks, on-page
›Site audit — technical + SEO audits vs. competitors
›Website health — Core Web Vitals, SSL, security, broken links
›GEO / AEO — visibility in ChatGPT, Gemini, Perplexity

Content & creative

Plan and produce client-facing work — every piece routed through the quality gate before it ships.

›Blog — research-driven SEO posts, per-client voice
›Social — platform-specific content
›Copywriter — ad / landing-page / email copy
›Content strategist — gaps, funnel mapping, distribution

Paid & channel

Deep channel analysis and the creative strategy that feeds it.

›Google Ads — search terms, quality scores, wasted spend
›Ad-creative strategist — testing strategy across Google + Meta

Platform operations

Keeps the platform itself healthy — separate from the client-facing work.

›Ops — system health, errors, cron + service monitoring

Quality & oversight

Grades the rest — nothing reaches a client unreviewed, and the agents themselves get performance-reviewed.

›Quality judge — scores every output 0–100 before it ships
›Weekly HR review — ranks agents, tunes the weakest one’s training

how they're trained

Each one is trained like an associate, not prompted like a chatbot

A director's answer is only as good as its weakest input. One specialist handing back something wrong or off-brand poisons the whole report. So each agent is engineered to a standard, not just given a prompt. Every one has a detailed expertise file, effectively an SOP: exactly what data to pull, the output format, the judgment calls, and what "good" looks like. I tuned each against a bar I'd set for a strong human associate, and spent real time testing them until the output matched what I'd actually hand a client.

Two mechanisms keep that bar from drifting. The quality judge grades every output before it ships. And once a week an HR-style review ranks the whole fleet by score, finds the weakest performer and the pattern behind it, and prescribes the exact change to that agent's expertise file, so the team improves under management instead of quietly degrading.

The weekly performance review

A scheduled "Quality Manager" pipeline pulls the last 7 days of quality scores and any rejected verdicts, ranks the agents, and writes an internal report: who's weakest, whether trends are improving or declining, and one specific edit to make to the lowest performer's training. The fleet is managed like staff, reviewed and coached rather than set and forgotten.

engineering highlights

The parts I'm proud of

Answers grounded in real data, by construction

The gather step forces tool_choice: 'any' on the first round, then switches to 'auto' once data is in hand. An agent physically cannot respond before it has pulled live numbers, which removes the hallucinated-metric failure mode that makes most analytics chatbots untrustworthy.

Fig. — pipeline (per query)

const pipeline = [
  { name: 'gather',  tools: agent.tools, toolChoice: 'any'  }, // must pull real data
  { name: 'respond', tools: [],          toolChoice: 'auto' }, // then synthesize
]
// round 1 forces a tool call; later rounds relax to 'auto' so it can answer

An LLM that grades the LLM

Before any output is trusted, a judge model (temperature 0) scores it against a weighted rubric and assigns a verdict. Accuracy auto-zeros on 3+ hallucinations; budget recommendations are tagged by risk tier. Scores persist, so I can watch each agent's quality trend and catch drift.

Fig. — quality rubric

accuracy      25%   // 3+ invented facts → auto-0
completeness  20%
actionability 20%
relevance     15%
quality       10%
excellence     5%
risk           5%   // budget tier: FREE / LOW / MED / HIGH

verdict = score >= 75 ? 'approved'
        : score >= 50 ? 'conditional'
        :               'rejected'

It spends real money, so it has a brake

Every run's token cost is estimated and accumulated against a daily budget. Cross the cap and the whole fleet is blocked until midnight; at 80% it warns. A runaway loop can't quietly burn the month's API spend.

Every tool call is an audit record

Each call writes a row to Postgres (agent, step, tool, arguments, a result preview, and duration), tied to a run ID. When an answer looks off, the exact chain of data calls behind it is fully reconstructable.

stack

Built with

Node (ESM) services · Anthropic SDK (Sonnet for agents, Opus for the judge) · Supabase / Postgres + pgvector · self-hosted Matrix (Conduit) chat backbone · Model Context Protocol · Railway (config-as-code, 5 services) · Sentry · healthchecks.io. Roughly 50K lines across the platform.

Client-specific details (account IDs, campaign names) are abstracted here; the architecture and mechanisms are shown as built.

← All projects