852 IT & Software Developer jobs in the US
Machine Learning Inference Architect
$132,000 - 222,100
eBay
W Parmer Lane 7700, Austin
$132,000 - 222,100
Requirements
Must:
- I have over 10 years of experience in building large-scale, fault-tolerant, high-performance distributed systems.
- I possess strong programming skills in one or more of the following languages: Java, Go, Rust, or C++ (Java is preferred for gateway services).
- I have a deep understanding of networking, concurrency, memory management, and performance tuning within production systems.
- I have proven experience in designing and operating low-latency APIs at very large scales (10M+ QPS).
- I am hands-on with Kubernetes, service meshes, and container orchestration at scale.
- I have a solid background in cloud infrastructure (AWS, GCP, Azure) and distributed system design.
Responsibilities
- I design and construct an LLM inference gateway that scales to billions of daily requests with millisecond-level latency.
- I develop intelligent request routing, load balancing, and fallback mechanisms across various LLM backends (both internal and external).
- I optimize throughput, cost, and reliability of inference workloads in multi-tenant environments.
- I collaborate with platform, research, and product teams to integrate new models and agentic capabilities into the gateway.
- I implement observability, tracing, and autoscaling for inference traffic across Kubernetes-based clusters.
- I conduct design and code reviews to uphold high standards in distributed systems architecture.
- I stay informed about advances in LLM serving, inference acceleration, and model APIs to continuously enhance the platform.
Description
At eBay, we see ourselves as more than just a global leader in ecommerce; we are transforming the way the world shops and sells. Our platform empowers millions of buyers and sellers across over 190 markets worldwide. We are dedicated to innovation and making a meaningful impact as we reshape the future of ecommerce for all. Our customers guide us; authenticity flourishes here, bold ideas are encouraged, and everyone is invited to express their individuality daily at work. We stand united in our mission to sustain the future for our customers, our company, and our planet. Join our team of passionate thinkers, innovators, and dreamers, and help us connect people and foster communities to create economic opportunities for everyone.
Something wrong or incorrect with this job? Tell us in the chat 💬 on the right ➡️
You can find Machine Learning Engineer salaries in the United States here.
How many Machine Learning Engineer jobs are in the United States?
Currently, there are 852 ML, AI openings. Check also: TensorFlow jobs, Python jobs, Computer-Vision jobs - all with salary brackets.
Is the US a good place for Machine Learning Engineers?
The US is one of the best countries to work as a Machine Learning Engineer. It has a vibrant startup community, growing tech hubs and, most important: lots of interesting jobs for people who work in tech.
Which companies are hiring for Machine Learning Engineer jobs in the United States?
Sperasoft, New World Technologies Inc., Giesecke+Devrient, Archon Systems Inc, Puter Technologies Inc., HAPLY Robotics Inc., LGS, une Société IBM / an IBM Company among others, are currently hiring for ML, AI roles in the United States.
The company with most openings is Leidos as they are hiring for 89 different Machine Learning Engineer jobs in the United States. They are probably quite committed to find good Machine Learning Engineers.
The company with most openings is Leidos as they are hiring for 89 different Machine Learning Engineer jobs in the United States. They are probably quite committed to find good Machine Learning Engineers.