Shkumbin Sherifi — AI Systems Engineer

Shkumbin
Sherifi

I build and harden AI agent systems for real operational workflows.From runtime governance and evaluation infrastructure to production reliability.

Tool & Policy Gateways · Agent Evals · Runtime Observability

A multi-tenant governed AI agent platform for HR and Campaign Operations, built around a central policy-enforcing execution gateway. The LLM proposes, the gateway decides, and the executor verifies.

Live synthetic scenarios
12/12
Policy eval cases
35/35
Tests passed · 4 skipped
196
Controlled autonomy

Server-side policy enforcement (role-rank, campaign role-family, scope, arguments, and quota) enforced in the gateway, not the prompt

Ed25519-signed execution tickets bind authorization to principal, tenant, role, tool, SHA-256 canonical arguments, and policy version; verified by public key before execution

Single-use replay protection with expiry and opt-in Postgres-backed distributed nonce support

ACL-aware RAG filtered by tenant and role at the query level, plus Supabase RLS isolation

Delivery & verification

12 live agent scenarios (incl. 3 policy-valid-but-harmful cases) + 35 deterministic policy cases + 29-trace grader validation (12 good, 17 bad)

End-to-end audit trail preserving ticket decision_id, arguments_hash, and non-silent audit-write failure flags

Provider failover: Gemini Flash → Cohere → offline mock

FastAPI · Next.js · Supabase · Ed25519

Applied AI · Workflow

Campaign Operations Copilot

Evidence-grounded campaign review with human authorization. Analyzes evidence, cites sources, detects contradictions and missing evidence, and prepares recommendations. Humans make the final decisions.

Eval fixtures · deterministic
30
Contradiction detection
1.00
Missing evidence recall
1.00
Citation correctness
0.875
Judgment stays human

Abstains on uncertainty instead of guessing

Injected instructions in untrusted evidence stay contained

Sensitive and cross-border cases escalate to specialists

Evidence

Deterministic fixtures with versioned results

Full audit trail on every review action

Tenant-isolated data access enforced by row-level security

Terminal-Bench style C++ benchmark for testing whether coding agents can solve a hidden 2D heat-equation task under private cases and shared runtime constraints.

Reference error
0.00113
Error gate
0.005
Replay baselines
3
Evaluation Harness

Docker verifier with hidden manufactured solutions

Trusted ADI/Crank-Nicolson numerical reference

Replay baselines for Codex, adversarial Codex, and Claude artifacts

Reference-margin audit strengthened the trusted solver without changing the agent-facing task

Result

Rejects coarse time stepping, explicit instability, and brute-force over-resolution

Prior pass artifacts replay to reward 0.0 while the trusted reference remains deterministic

Fresh Claude run produced a failing partial artifact before hitting provider rate limits

Contact

Open to AI systems, ML infrastructure, and applied AI engineering roles.