State Street is looking for a QA Engineer to join our Artificial Intelligence initiatives. You will work within a multidisciplinary team focused on emerging trends and their application in financial services.
What You'll Do
- Analyze business and technical specifications.
- Write functional test cases and perform regression testing.
- Report bugs, verify fixes, and maintain the bug tracking system.
- Maintain and execute automated tests using Selenium with Python.
- Create automated test scripts using the Pytest framework.
- Automate web services using SoapUI and/or Python.
- Design and execute test plans for Large Language Models, including prompt validation, accuracy checks, and safety compliance.
- Perform Agents testing for AI-driven workflows to ensure reliability, robustness, and integration quality.
What We're Looking For
- 8+ years of software quality assurance testing experience.
- Experience with tools like PyCharm, Visual Studio Code, Git, Fiddler, Postman, SoapUI, JMeter, Jenkins, and Jira.
- Experience in CI/CD-based projects.
- Ability to understand customer needs and translate them into concepts and prototypes.
- Strong analytical, technical, and problem-solving skills.
- Ability to manage your own time by prioritizing tasks within deadlines.
- Experience with BDD & TDD using Gherkin.
- Experience using Jira as an agile management tool.
- Experience using selenium with python for test automation.
- Experience in testing Large Language Models, including validation of prompt responses and automation of LLM-based workflows.
- Red teaming of LLMs to identify vulnerabilities and edge cases through adversarial prompts.
- Safety validation and ethical compliance testing of LLM behavior.
- Familiarity with tools and frameworks for LLM evaluation such as OpenAI Evals, Ragas, and custom benchmarking suites.
Nice to Have
- Hands-on experience in LLM testing and Agents testing.
- A basic understanding of artificial intelligence and machine learning algorithms.
- Familiarity with AI governance, bias detection, and safety validation.
Technical Stack
- Python, Selenium, Pytest, SoapUI, Jira, PyCharm, Visual Studio Code, Git, Fiddler, Postman, JMeter, Jenkins
Work Mode
This role follows a hybrid work model.
As an Equal Opportunity Employer, we consider all qualified applicants for all positions without regard to race, creed, color, religion, national origin, ancestry, ethnicity, age, disability, genetic information, sex, sexual orientation, gender identity or expression, citizenship, marital status, domestic partnership or civil union status, familial status, military and veteran status, and other characteristics protected by applicable law.
