Shkumbin Sherifi — AI Systems Engineer

Shkumbin
Sherifi

I build and harden AI agent systems for real operational workflows, from infrastructure and evaluation to observability, guardrails, and production.

AI agent systems · Agent infrastructure · Integration · Evaluation · Observability · Guardrails · Production

View Systems Contact

A production-style governed AI platform running agents in two operational environments (HR workflows and Support Operations) on one gateway. Same guardrails, same audit trail: the LLM proposes, the gateway decides.

Risk tiers: Auto · Approval · Block: 3
Eval assertions · CI: 59
Deployed API + web: Live

Controlled autonomy

Risk-tiered tool authorization (auto / approval / hard-blocked), enforced in the gateway, not the prompt

Approval workflow re-runs the full policy check before anything executes

ACL-aware RAG filtered by tenant and role at the query level, plus cross-tenant isolation

End-to-end audit: every turn, tool decision, approval, and remediation traced

Delivery & verification

Deterministic eval suite (16 scenarios) incl. adversarial; runs in CI on every push

Provider failover: ox-alpha → Cohere → offline mock

FastAPI on Render, Next.js on Vercel, deployed from main via the render.yaml blueprint

Multi-tenant isolation: Supabase RLS on every tenant-scoped table

Evidence-grounded campaign review with human authorization. Analyzes evidence, cites sources, detects contradictions and missing evidence, and prepares recommendations. Humans make the final decisions.

Eval fixtures · deterministic: 30
Contradiction detection: 1.00
Missing evidence recall: 1.00
Citation correctness: 0.875

Judgment stays human

Abstains on uncertainty instead of guessing

Injected instructions in untrusted evidence stay contained

Sensitive and cross-border cases escalate to specialists

Evidence

Deterministic fixtures with versioned results

Full audit trail on every review action

Tenant-isolated data access enforced by row-level security

Terminal-Bench style C++ benchmark for testing whether coding agents can solve a hidden 2D heat-equation task under private cases and shared runtime constraints.

Reference error: 0.00113
Error gate: 0.005
Replay baselines: 3

Evaluation Harness

Docker verifier with hidden manufactured solutions

Trusted ADI/Crank-Nicolson numerical reference

Replay baselines for Codex, adversarial Codex, and Claude artifacts

Reference-margin audit strengthened the trusted solver without changing the agent-facing task

Result

Rejects coarse time stepping, explicit instability, and brute-force over-resolution

Prior pass artifacts replay to reward 0.0 while the trusted reference remains deterministic

Fresh Claude run produced a failing partial artifact before hitting provider rate limits

2026-07-156 min read

How I Beat Anthropic's 1,487-Cycle Benchmark with Claude and Codex

A human-directed, AI-assisted optimization session reached 1,469 cycles on Anthropic's open performance challenge while preserving correctness and keeping the implementation private.

Performance EngineeringClaudeCodexAI Agents

2026-07-099 min read

Breaking into Y Combinator: Part 3 | The Hardening Loop That Made Codex Fail

Designing and hardening a 2D C++ heat-diffusion Terminal-Bench task with hidden manufactured solutions, and the hardening loop that produced the first clean Codex failure.

TB3Benchmark DesignNumerical MethodsAgent Evaluation

2026-08-244 min read