A continuously improving, experimentation framework.
npm i @bkknights/prosperyarn add @bkknights/prosperProsper provides a means of:
- injecting intelligently selected experimental code that is shorted lived
- using multi armed bandit machine learning to selected which experimental code is injected
- prevents code churn where long-lived code belongs
The non-Prosper way:
- Uses feature flagging
- Favors code churn, with highly fractured experimentation
- Constantly effects test coverage
- Provides a very blurry understanding of the code base when experimenting
The Prosper way:
- Use experiments rather than Features Flags
- Picture one master switch, rather than a many small switches
- Code for each variant lives close together, within an experiment
- Favors short-lived experimental code, that accentuates long-lived code
- Once understandings from a variant is known, then it can be moved from short-lived (experiment) to long-lived (source)
- Meant to churn as little as possible, using decorator
@pick(symbol)with class properties. - Provides a very clear understanding of the code base when experimenting
This is considered 'long-term' code. Our goal is that we want to find out if a Vulcan reply string is better, based on some user interaction.
class ReplyHandler { reply() { return 'See ya!'; } } class Service { replyHandler: ReplyHandler = new ReplyHandler(); async setup(httpEvent): Promise<void> {} run(): string { return this.replyHandler.reply(); } }Using Feature Flags, and a hypothetical function findBestReplyIndex, which finds the best index for user. Note: We now have to change from sync to async, which changes usage as well.
class ReplyHandler { async reply(): string { // short lived code const index = await findBestReplyIndex(); // long lived code const defaultValue = 'See ya!'; // short lived code if (featureFlagsEnabled()) { // If's can get tricky, when experimenting... switch (index) { case 0: return 'Live long, and prosper.'; // Others? } } return defaultValue; } } class Service { replyHandler: ReplyHandler = new ReplyHandler(); async setup(httpEvent): Promise<void> {} // Note: Changed from sync to async async run(): string { return await this.replyHandler.reply(); } }Using Experiments
// Imports import { BaseExperiment, Variant } from '@bkknights/prosper'; // Create a symbol to reference ReplyHandler const replyHandlerSymbol = Symbol('ReplyHandler'); // class Experiment extends BaseExperiment { // pretend I've connected it up to database } // Setup short lived experimentation Code function whichReply(): Experiment { class VulcanReply extends ReplyHandler { reply(): string { return 'Live Long And Prosper'; } } // Setup both defaults, and control set against experiment return new Experiment('Which Reply Is Best?', [ new Variant('A: Control Set', { [replyHandlerSymbol]: ReplyHandler }), new Variant('B: Vulcan Greeting/Reply', { ...defaults, [replyHandlerSymbol]: VulcanReply }), ]); } // Instantiate Prosper, with whichReply experiment set const prosper = new Prosper().with(whichReply()); class Service { prosper = prosper; // Note: Or inject? // Note: Now "picking" from multiple ReplyHandler's, associated in setup @pick(replyHandlerSymbol) replyHandler: ReplyHandler; // Note: Need to allow prosper to both setup and bind to a value that persists over time in 1 key location within codebase async setup(httpEvent): Promise<void> { await this.prosper.setForUser(httpEvent.userId); } // Note: usage stayed the same run(): string { return this.replyHandler.reply(); } }- Tests remain isolated, period.
- A/B tests are very focused and isolated
Interacting with Prosper is done by creating a single instance of prosper used on classes where @pick(Symbol) is used.
import { Prosper, pick } from '@bkknights/prosper'; const prosper = new Prosper(); class MyClass { prosper: Prosper = prosper; @pick(Symbol('foo')) foo: Function; bar() { this.foo(); } }Prosper is interacted with by extending the abstract class BaseExperimentSet
import { Prosper, pick } from '@bkknights/prosper'; import { BaseExperiment } from '@bkknights/prosper/base-experiment'; export class MyExperiment extends BaseExperiment<AlgorithmType> { public async getExperiment(): Promise<IExperiment | null> { } public async upsertExperiment(experiment: IExperiment): Promise<void> { } public async deleteExperiment(experiment: IExperiment): Promise<void> { } public async getUserExperiment(userId: string, experimentId: string): Promise<IUserVariant | null> { } public async upsertUserVariant(userVariant: IUserVariant): Promise<void> { } public async deleteUserVariant(userExperiment: IUserVariant): Promise<void> { } public async deleteUserVariants(): Promise<void> { } public async getAlgorithm(): Promise<Algorithm> { } public async upsertAlgorithm(algorithm: Algorithm): Promise<void> { } public async deleteAlgorithm(): Promise<void> { } public async getVariantIndex(algorithm: Algorithm): Promise<number> { } public async rewardAlgorithm(algorithm: Algorithm, userVariantIndex: number, reward: number): Promise<Algorithm> { } } new Prosper().with(setupEvents(new MyExperiment()))Variants are written and added to an MyExperiment
import { Prosper } from '@bkknights/prosper'; import { BaseExperiment } from '@bkknights/prosper/base-experiment'; import { Variant } from '@bkknights/prosper/variant'; const fooSymbol = Symbol('foo'); const foo1 = () => { // do default }; const foo2 = () => { // do experiment! }; class Experiment extends BaseExperiment { constructor(name: string, variants: Variant[]) { super(); this.name = name; this.variants = variants; } } const prosper = new Prosper() .with( new Experiment('My Experiments', [ new Variant('Control Set: A', { [fooSymbol]: foo1 }), new Variant('Deveation: B', { [fooSymbol]: foo2 }), ]) ); // call and `await prosper.setForUser(key)` just after database connectivity! // elsewhere in codebase class MyClass { prosper = prosper; @pick(fooSymbol) foo: Function; myMethod() { this.foo(); // calls either `foo1` or `foo2`, whichever the algorithms is indicating } }...Vulcan's are cool.