Prompt Studio is still in development, and we're looking for feedback! Sign up for free and try it out. Let's shape the future of prompt engineering together.
- One user
- Unlimited test runs
- Up to 10 projects
- Up to 15 variants per project
- Up to 20 test suites per project
- Connects with OpenAI API
- Prompt security testing
- 5 users
- Unlimited test runs
- Up to 100 projects
- Up to 50 variants per project
- Up to 100 test suites per project
- Connects with OpenAI, MS Azure, Claude and Mistral APIs
- Connects with your own API
- Prompt security testing
- Custom plan tailored to your needs
Lyris Prompt Studio embraces test-driven development, a well-known concept in software engineering. Test-driven development (TDD) places the focus on clearly defining the desired behaviour of a prompt. Unlike before, prompt performance becomes measurable for the first time. TDD means a focus on reliability and consistency.
Setting clearly defined, measurable objectives
- Define your objective: Define the desired behaviour of your prompt through quantifiable tests. Tests provide an objective measure of your prompt's performance in Prompt Studio and allow you to ensure that your prompt does what you expect it to do.
- Measure performance: Run your prompt against your tests as often as needed, and view the results in an aggregated, easy-to-understand overview. This gives you a clear picture at a glance, but also allows you to drill down into the details to find out why specific tests might have failed.
The takeaway
Lyris Prompt Studio reorients your workflow towards target-oriented development and away from trial and error, through test-driven development. Tests let you define a measurable foundation for your prompt's performance and reduce inconsistency and complexity.
Lyris Prompt Studio takes the anxiety out of deploying updated prompts. By leveraging quantifiable test results, you can compare different prompt variants and ensure that everything that worked before still works - and your new features, too. And in case it doesn't, you'll know exactly what's wrong.
Safe deployment through test result comparison
- Compare performance: Compare the performance of a new prompt variant with the production variant in Prompt Studio, making sure that everything works as expected. This way, you make sure that no existing functionality was lost when making changes to your prompt.
- Deploy with confidence: With the assurance that your prompt's performance is consistent with the production variant, you can deploy your new prompt variant with confidence. No surprises, everything has been tested repeatedly.
The takeaway
It's the end of deployment anxiety. Lyris Prompt Studio uses its test-driven development approach to measure your prompt update's performance and compares it to the production variant. This gives you the confidence to deploy at just a glance. This also allows you to try out different variants of your prompt, and pick the optimal one.
Many prompts are vulnerable to attacks if they contain user input. Malicious users might be able to attack your app through unsafe prompts. Lyris Prompt Studio can automatically test your prompts for common vulnerabilities, minimizing the risk of potentially harmful attacks.
Built-in security testing
- Proactive security measures: Prompt Studio offers built-in security testing for common prompt vulnerabilities, including:
- Prompt injection: By injecting additional instructions into your prompt, attackers can attempt to manipulate your app's behaviour.
- Prompt leakage: Attackers can attempt to extract hidden instructions in your prompt, potentially revealing implementation details of your app.
- Pre-configured tests: Prompt Studio comes with pre-configured security test, so you don't have to worry about constructing common attack scenarios yourself.
The takeaway
Don't let your app's integrity be compromised. Lyris Prompt Studio offers built-in security testing for prompt vulnerabilities, with out-of-the-box tests for common attack scenarios. Test your prompts for attack vectors before attackers do.
LLM outputs are notoriously inconsistent. While a prompt might produce the desired output once, it might not do so the next time. Lyris Prompt Studio uses its test-driven development approach paired with repetition testing to reduce the impact of this output dispersion when assessing your prompt's performance.
Reducing the impact of output dispersion
- Repetition testing: Prompt Studio lets you run your tests multiple times and aggregates the results in an intuitive overview. This considerably reduces the impact of output dispersion and gives you a performance measurement that matches real-world usage more closely.
- Performance drill-down: When your prompt fails a test, you can drill down into the details of each individual run to find out why it failed, with detailed explanations for each individual test and repetition. Additionally, aggregated views let you quickly discover weak spots in your prompt.
The takeaway
Reducing the impact of output dispersion significantly, Lyris Prompt Studio enables you to measure your prompt's performance in the real world. Aggregated performance results from repetition testing give you a clear picture of your prompt's consistency and performance at a glance, while drill-down inspection lets you find even the smallest weak spots in your prompt.
Building a prompt with variable parts and user input quickly becomes a headache when it comes to managing costs and and the context window limit. Lyris Prompt Studio lets you configure your prompt's variables and automatically calculates token usage and costs in real time - both when developing your prompt, and when setting up tests for it.
Keep track of usage as you go
- Token usage: Prompt Studio calculates the token usage of your prompt, including custom variables. This way, you can make sure not to exceed the model's context window limit.
- Cost calculation: Stay on top of your prompt's cost by monitoring the real-time cost of inference. Prompt Studio calculates the cost automatically for you, and breaks it down by part of your prompt.
The takeaway
No more Excel sheets or manual token calculations. Lyris Prompt Studio shows you token usage and inference cost for your prompt in real time, as you develop it. It also intelligently reduces the cost of testing to save you money on every test run, making it cheaper than manual testing.
BY CODIAC KNOWLEDGE ENGINEERING GMBH