1080 IT & Software Developer jobs in the US
LLMOps Engineer - Operations & Quality Assurance
$70,000 - 80,000
Acti
From Road 650, Paramus
$70,000 - 80,000
Requirements
Must:
- 3+ years of experience in DevOps, SRE, MLOps, or a similar operations-oriented engineering role
- Strong proficiency in Python with experience in developing monitoring and observability systems
- Familiar with logging and monitoring tools like Datadog, Grafana, Prometheus, CloudWatch, or similar
- Understanding of LLM APIs, token-based pricing structures, and architectures of AI systems
- Experience in establishing evaluation frameworks, testing pipelines, or quality assurance systems for software products
- Knowledgeable about CI/CD tools and deployment automation methods, such as GitHub Actions, Jenkins, or equivalent
- Strong analytical abilities to identify data patterns and convert them into actionable recommendations
- Direct exposure to LLMOps tools (LangSmith, Weights & Biases, Humanloop, or similar) is a plus
- Experience in managing costs and optimizing resource consumption for API-intensive systems
- Background in creating dashboards and data visualization (Metabase, Looker, custom solutions) is advantageous
- Familiarity with prompt engineering and understanding the impact of prompt variations on model behavior
- Experience with multi-agent systems or orchestration platform monitoring is beneficial
- Knowledge of AI safety, content moderation, and ethical AI deployment practices is preferred
Responsibilities
- Design and implement detailed monitoring and observability systems for all operational AI agents, tracking response quality, latency, error rates, and conversation results
- Build and uphold evaluation frameworks to assess agent performance against established benchmarks, including automated quality scoring and regression identification
- Oversee token utilization, API expenses, and resource distribution across all agents and LLM suppliers, providing regular cost analysis and optimization suggestions
- Develop and maintain robust conversation logging infrastructure for analysis, debugging, and compliance purposes
- Implement hallucination detection, content safety filters, and guardrail systems to safeguard end users and uphold brand integrity
- Create and manage alerting mechanisms for agent failures, performance declines, and unusual behavior patterns
- Establish A/B testing and prompt version control systems to assist the Prompt Architect in ongoing agent enhancements
- Create and maintain CI/CD pipelines for prompt releases, ensuring testing, staging, and safe rollouts of changes
- Develop dashboards and reporting tools to provide leadership with insights into agent performance, return on investment (ROI), and operational health
- Collaborate with the AI/ML Engineer on infrastructure enhancements and with the Solutions Engineer to ensure production reliability
Description
At Actualized Intelligence (ACTi), we manage an expanding array of AI agents that engage in crucial conversations—from legal outreach to personal coaching. The efficacy, dependability, and cost-effectiveness of these agents play a vital role in our success and our clients results. We are seeking an LLMOps Engineer to take charge of the operational integrity and ongoing enhancement of our entire agent ecosystem. We offer a hybrid remote work environment based in Paramus, NJ, with competitive salaries ranging from $70,000 to $80,000 annually and attractively comprehensive benefits including health, dental, and vision insurance, as well as a 401(k) plan and paid time off.
Something wrong or incorrect with this job? Tell us in the chat 💬 on the right ➡️
You can find QA, Test Engineer salaries in the United States here.
How many QA, Test Engineer jobs are in the United States?
Currently, there are 1080 QA, Test openings. Check also: Selenium jobs, Cypress jobs, Cucumber jobs - all with salary brackets.
Is the US a good place for QA, Test Engineers?
The US is one of the best countries to work as a QA, Test Engineer. It has a vibrant startup community, growing tech hubs and, most important: lots of interesting jobs for people who work in tech.
Which companies are hiring for QA, Test Engineer jobs in the United States?
LYNKED Inc., D3 Security Management Systems, Snaplii, Frontier Dental Supply, DataAnnotation, Quantic, Alyn Inc. among others, are currently hiring for QA, Test roles in the United States.
The company with most openings is Jobot as they are hiring for 196 different QA, Test Engineer jobs in the United States. They are probably quite committed to find good QA, Test Engineers.
The company with most openings is Jobot as they are hiring for 196 different QA, Test Engineer jobs in the United States. They are probably quite committed to find good QA, Test Engineers.