Software Engineer – AI Evaluation Expert
About The Role What if your engineering instincts could directly shape how the world's most advanced AI systems write and reason about code? We're looking for experienced software engineers to evaluate and improve frontier AI models — using your real-world expertise to catch what others miss: subtle bugs, flawed logic, hallucinated outputs, and edge cases that only a seasoned engineer would spot. This is a fully remote, flexible contract role. No AI research background required — just deep, battle-tested software engineering skill and a sharp eye for what good code actually looks like.
- Organization: Alignerr
- Type: Hourly Contract
- Location: Remote
- Commitment: 10–40 hours/week
What You'll Do
- Evaluate the performance of frontier AI language models on complex, real-world software engineering tasks
- Identify bugs, logical errors, hallucinations, and reliability issues in model-generated code and explanations
- Design and review prompts, test cases, and evaluation scenarios that stress-test advanced coding workflows
- Write precise, structured feedback that clearly explains model strengths, weaknesses, and failure modes
- Work across multiple languages and codebases to assess how well AI generalizes across different engineering contexts
Who You Are
- 3–4+ years of professional software engineering experience
- Strong proficiency in at least one of: TypeScript, Ruby, Java, or C++
- Excellent written and spoken English — you can explain complex technical reasoning clearly
- Able to reason about systems deeply and debug non-obvious issues with confidence
- Familiar with modern development workflows — Git, CLI tools, testing frameworks, and the like
- You critically evaluate code and model outputs rather than simply accepting them at face value
Nice to Have
- Experience working with AI tools, LLMs, or prompt engineering
- Background in code review, technical writing, or software quality assurance
- Exposure to multiple programming languages and diverse codebases
- Familiarity with evaluation methodologies or red-teaming practices
Why Join Us
- Work on cutting-edge AI projects alongside leading research labs
- Fully remote and flexible — work when and where it suits you
- Freelance autonomy with the structure of meaningful, task-based engineering work
- Make a direct, tangible impact on how AI understands and generates software
- Potential for ongoing work and contract extension as new projects launch
Apply tot his job Apply To this Job