Red Hat is looking to test and validate their Deep Learning Execution Engine and inference optimization products, specifically focusing on the performance and accuracy of Large Language Models (LLMs) for vLLM and llama.cpp inference on different accelerators.
Requirements
- Solid experience evaluating LLMs for performance on accelerators and accuracy (think HellaSwag, MMLU, Chatbot Arena, TruthfulQA, etc.).
- Being super comfortable with Python and PyTest is a must.
- Familiarity with Git, GitHub, or GitLab.
- Strong experience with API and performance testing, especially for C++ and Python.
- You should be a pro with Docker, Podman, and Kubernetes or Openshift.
- Highly experienced in setting up CI/CD processes like Jenkins and GitHub Actions.
- Understanding of core Machine Learning algorithms and basics
Responsibilities
- Testing the performance and accuracy of LLMs for vLLM and llama.cpp inference on different accelerators.
- Making awesome test plans and cases to hit product requirements.
- Doing all sorts of testing: functional, performance, regression, you name it, to make sure the product is solid.
- Writing test code and frameworks to automate testing.
- Monitoring, analyzing, and reporting test results and failures.
- Sharing your knowledge and recommendations to help the team keep getting better.
- Giving good and quick code reviews.
Other
- At least 3 years of software testing experience.
- Knowing how to excel in Open Source communities.
- A track record of contributing to the vLLM community is a huge plus!
- The salary range for this position is $133,650.00 - $220,680.00.
