Adem Garic
QA Engineer, AI · SDET · QA Analyst
I build automation frameworks, AI evals, and QA agents. I also break things on purpose. That covers six years of web, mobile, and API testing and, lately, grading LLM output where there's no single right answer.
// about
Different worlds, same instinct: find the detail everyone else missed.
I started in manual and exploratory testing: writing test cases, triaging defects, and learning how real users break software. From there I moved into SDET work. I built an Appium 2 mobile framework from scratch for iOS and Android, maintained business-critical Selenium suites, led the move to Playwright, and wired it all into Jenkins, Bitrise, and BrowserStack.
Today I work in AI quality as a QA Engineer, AI at TrueLearn. I build test sets and evaluation systems for AI features, write LLM-as-a-judge graders, and hunt for hallucinations, retrieval regressions, and guardrail breaks. On the side I build my own AI testing tools: MCP servers, Claude-powered QA agents, and eval harnesses.
I also shipped my own AI product, Pitchlio, where I was the PM, the developer, and the QA engineer. Outside engineering I'm a published author and a literary translator between Bosnian and English. That work trains the same eye I bring to testing.
01Risk first, coverage second
I test where failure costs the most. A long list of green checks can still miss the one flow that matters.
02“Flaky” is a finding, not an excuse
Many of the bugs I've found were hiding behind what everyone else had written off as flakiness.
03Test the judge before you trust it
An eval is only as good as its grader. I build grounded and adversarial cases first, so I know every right answer before the model runs.
04Bug reports are a product
A report should be clear, reproducible, and evidence-based, so a developer can act on it without a follow-up call.
// pick a lens
Three kinds of QA, one engineer.
Pick the role you're hiring for to see the relevant skills, evidence, and tools.
// interactive · ai quality · case study
Eval Lab: can an AI résumé writer be trusted?
My product Pitchlio tailors résumés with Claude. A tailored résumé can be persuasive, well written, and false: “mentored” becomes “led”, a tool from the job posting slips into a real bullet, a number grows. These are the evals I built to catch that. Start by doing the judge's job yourself.
01Try it: label each claim
Here's a short source résumé for a made-up SDET and five claims from a résumé tailored to a job posting. Give each claim one of the five labels, then run the judge.
Source résumé (made-up person)
- Mentored two junior QA engineers on Selenium best practices.
- Built a Jenkins pipeline that runs 300 regression tests nightly.
- Reduced flaky test failures by 30% by fixing waits and locators.
- Wrote API tests with RestAssured for the payments service.
The policy
| supported | Stated in the source résumé | OK |
| reworded | Same fact in the posting's words | OK |
| overstated | Real but inflated | Fail |
| unsupported | No basis in the source | Fail |
| contradicted | Conflicts with the source | Fail |
02The real thing: Pitchlio's eval suite
The same idea at scale: test the judge first, then use it to test the generator.
Deterministic checks
Flags numbers, dates, titles, employers, and skills that aren't in the source résumé, plus format rules. No AI, runs in CI.
AI judge (Claude Opus)
Splits each document into single claims and labels each one against the source. Catches what text matching can't.
The judge is a different model from the generator (Sonnet), so nothing grades its own writing. Where the line sits (reworded is fine, overstated fails) is a written product decision, not left to the model. And an error is not a fail: refusals and API failures are counted separately.
lies caught, 0 false alarms. Missed every seniority, responsibility, and keyword-stuffing lie.
lies caught, and 0 of 6 honest résumés wrongly flagged.
of verdicts identical across both runs.
Testing the test: 24 hand-labelled résumés for two made-up people, 6 honest and 18 with exactly one planted lie from 13 kinds. Each case records which layer should catch it. Only after this did I trust the judge on real output.
in every tailor run so far, made-up people and my real résumé alike.
across 10 tailor runs, never 100%.
must-fix and should-fix items, résumé and cover letter together.
The generator was the weak spot, not the judge. The first baseline exposed root causes in my own prompt design:
- Anecdotes from the writing sample, meant only to match voice, leaked into cover letters as facts.
- The “Problem, Action, Result” structure pushed the model to invent backstory.
- Most common today: cover-letter claims that aren't in the résumé (9 of 9 tailorings).
- A Fact check button on every tailored job.
- A review page where you chat with a reviser to fix flagged claims.
- Every revision is a saved version with free checks and a diff.
- “I did use Docker there” is saved as a confirmed fact. Unbacked requests get pushback.
3 revised applications, latest re-check. Small sample, stated as such.
| Judge setting | Lies caught | False alarms | Time / call | Cost |
|---|---|---|---|---|
| Full output, high effort | 11 of 11 | 0 of 4 | 17 s | $0.81 |
| Trimmed, high | 11 of 11 | 0 of 4 | 11 s | $0.57 |
| Trimmed, medium (default) | 11 of 11 | 0 of 4 | 9 s | $0.50 |
Every optimization was an experiment against the same labelled set, not a guess: about half the time and 40% less cost, with no loss in accuracy. The evals also measured an assumption nobody had checked: prompt caching hits 75–100% of the time. Measured cost is about $0.03–0.07 per tailoring and about $0.15 per real fact check.
03Six things I'd do again
Test the test
An LLM judge is a test oracle. Validate it on planted errors before trusting a single verdict.
Cheap checks first
Free checks caught 61% of lies with no AI. Spend model calls only on what text matching can't see.
Write the policy down
“Reworded is fine, overstated fails” is a product call. Make it explicit and shared.
Error is not fail
When API credits ran out mid-run, every case errored. Counted as fails, the report would have lied.
Act on claims, not decimals
Scores depend on how claims are split. The flagged sentences are what you fix.
Measure assumptions
Caching, cost, and speed claims went from “should work” to measured numbers.
The method started in my open-source faithfulness-evals project. Synthetic fixtures run in CI; real résumés and results stay local.
// manual qa · how I document
The artifacts behind the automation.
Good QA starts before any code: clear bug reports, sharp test cases, and a risk-based plan. These samples show the format and depth I work to.
iOS: Checkout “Pay” button stays disabled after saved card is re-selected when returning from 3-D Secure
Summary
After the user backs out of the 3-D Secure challenge and re-selects the same saved card, the Pay button never re-enables. The user can't complete the purchase without restarting the app.
Steps to reproduce
- Log in as
qa+3ds@example.com, add any item to the cart - Go to Checkout and select saved card
•••• 3220(3DS-enrolled test card) - Tap Pay; the 3DS sheet appears
- Tap Cancel on the 3DS sheet
- Tap a different card, then re-select
•••• 3220
Expected
Pay button becomes enabled once a valid card is selected.
Actual
Pay button stays disabled (isEnabled=false in the Appium inspector). No console error.
Evidence and notes
- Reproduces 5/5 on iOS 17.5 (iPhone 15, BrowserStack) and 3/3 on a physical iPhone 12
- Not reproducible on Android 14; this points at iOS-specific state handling
- Screen recording and network HAR attached. The
/payment-intentscall is never re-issued after cancel - Regression: first build affected is
4.18.0 (412);4.17.2is fine
- Priority
- P1: blocks revenue flow
- Env
- iOS 17.5 · Staging
- Build
- 4.18.0 (412)
- Component
- Checkout / Payments
- Repro rate
- 100% (8/8)
- Labels
- regression, ios, 3ds
- Automation
- Appium test added:
CheckoutTest#threeDsCancelReselect
Test cases: authentication (excerpt)
| ID | Scenario | Steps / data | Expected result | Type |
|---|---|---|---|---|
| TC-AUTH-01 | Valid login | Valid email + password → Sign in | Lands on dashboard; session cookie set with Secure; HttpOnly | smoke |
| TC-AUTH-04 | Invalid password, lockout threshold | Wrong password ×5 | Generic error (no user enumeration); 6th attempt locked for 15 min | negative |
| TC-AUTH-07 | Session persists across reload | Log in → hard reload → back/forward | User stays authenticated; no flash of login screen | regression |
| TC-AUTH-09 | Unauthenticated deep link | Open /secure/settings logged out | Redirect to login, then return to the original URL after sign-in | regression |
| TC-AUTH-12 | Whitespace and Unicode in email | " User@Exämple.com " | Trimmed, case-insensitive match; IDN handled or rejected with a clear message | edge |
| TC-AUTH-15 | Concurrent sessions | Log in on device A, then B; log out on A | Behaves per spec (single vs. multi-session); B's token not silently invalid | edge |
Risk-based plan: AI-generated study explanations
Hallucinated facts
Golden dataset + claim-level faithfulness judge on every prompt/model change. Release gate: 0 unsupported claims on the critical set.
Guardrail breaks
Adversarial set covering prompt injection, off-topic, and unsafe advice. Must refuse or redirect every time.
Retrieval regressions
Track context precision/recall on fixed queries. Alert when a re-index drops the right chunk.
Difficulty drift
Rubric-graded sample per release, checking that explanations match the learner level. Human spot-check on disagreements.
UI / API plumbing
Standard automated coverage in CI: UI, API contract, and service layer. Smoke on every PR.
Go / no-go
All high-risk gates green, no open S1/S2 defects, and judge agreement with human labels ≥ 90% on the calibration set.
* Illustrative samples written for this portfolio. Real client and employer artifacts are confidential.
// projects · open source
Things I've built to test things.
Every repo has a real problem statement and runnable code, and the README documents what broke along the way.
// experience
Where I've shipped quality.
Click a role to expand it.
// peer review
What colleagues say.
Nine LinkedIn recommendations, grouped by how we worked together. The excerpts are quoted word for word.
// credentials
Education & certifications.
Education
- M.S., Information TechnologyUniversity of the People · 2025 – present
- B.A., TheologyUniversity of Sarajevo · 2006 – 2011
AI & testing certifications
- Prompt Engineering SpecializationVanderbilt University · 2024
- Trustworthy Generative AIVanderbilt University
- ChatGPT Advanced Data AnalysisVanderbilt University · 2024
- AI-Powered TestingTecno Study · 2023
More
- LambdaTest Software Testing ProfessionalLinkedIn Learning · 2024
- BDD with Cucumber & SerenitySerenity Dojo · 2023
- CompTIA CySA+LinkedIn Learning · 2023
- LanguagesEnglish (fluent) · Bosnian (native) · Arabic (elementary)