A BVar s a is a value of type a that can be "backpropagated".

Functions referring to BVars are tracked by the library and can be automatically differentiated to get their gradients and results.

For simple numeric values, you can use its Num, Fractional, and Floating instances to manipulate them as if they were the numbers they represent.

If a contains items, the items can be accessed and extracted using lenses. A Lens' b a can be used to access an a inside a b, using ^^. (viewVar):

(^.) :: a -> Lens' a b -> b (^^.) :: BVar s a -> Lens' a b -> BVar s b 

There is also ^^? (previewVar), to use a Prism' or Traversal' to extract a target that may or may not be present (which can implement pattern matching), ^^.. (toListOfVar) to use a Traversal' to extract all targets inside a BVar, and .~~ (setVar) to set and update values inside a BVar.

If you have control over your data type definitions, you can also use splitBV and joinBV to manipulate data types by easily extracting fields out of a BVar of data types and creating BVars of data types out of BVars of their fields. See Numeric.Backprop for a tutorial on this use pattern.

For more complex operations, libraries can provide functions on BVars using liftOp and related functions. This is how you can create primitive functions that users can use to manipulate your library's values. See https://backprop.jle.im/08-equipping-your-library.html for a detailed guide.

For example, the hmatrix library has a matrix-vector multiplication function, #> :: L m n -> R n -> L m.

A library could instead provide a function #> :: BVar (L m n) -> BVar (R n) -> BVar (R m), which the user can then use to manipulate their BVars of L m ns and R ns, etc.

See Numeric.Backprop and documentation for liftOp for more information.

An ephemeral Wengert Tape in the environment. Used internally to track of the computational graph of variables.

For the end user, one can just imagine Reifies s W as a required constraint on s that allows backpropagation to work.

Class of values that can be backpropagated in general.

For instances of Num, these methods can be given by zeroNum, addNum, and oneNum. There are also generic options given in Numeric.Backprop.Class for functors, IsList instances, and Generic instances.

instance Backprop Double where zero = zeroNum add = addNum one = oneNum 

If you leave the body of an instance declaration blank, GHC Generics will be used to derive instances if the type has a single constructor and each field is an instance of Backprop.

To ensure that backpropagation works in a sound way, should obey the laws:

identity

• add x (zero y) = x

• add (zero x) y = y

Also implies preservation of information, making zipWith (+) an illegal implementation for lists and vectors.

This is only expected to be true up to potential "extra zeroes" in x and y in the result.

commutativity

• add x y = add y x

associativity

• add x (add y z) = add (add x y) z

idempotence

• zero . zero = zero

• one . one = one

unital

• one = gradBP id

Note that not all values in the backpropagation process needs all of these methods: Only the "final result" needs one, for example. These are all grouped under one typeclass for convenience in defining instances, and also to talk about sensible laws. For fine-grained control, use the "explicit" versions of library functions (for example, in Numeric.Backprop.Explicit) instead of Backprop based ones.

This typeclass replaces the reliance on Num of the previous API (v0.1). Num is strictly more powerful than Backprop, and is a stronger constraint on types than is necessary for proper backpropagating. In particular, fromInteger is a problem for many types, preventing useful backpropagation for lists, variable-length vectors (like Data.Vector) and variable-size matrices from linear algebra libraries like hmatrix and accelerate.

Since: 0.2.0.0

A newtype wrapper over an f a for Applicative f that gives a free Backprop instance (as well as Num etc. instances).

Useful for performing backpropagation over functions that require some monadic context (like IO) to perform.

Since: 0.2.1.0

A newtype wrapper over an instance of Num that gives a free Backprop instance.

Useful for things like DerivingVia, or for avoiding orphan instances.

Since: 0.2.1.0

Turn a function BVar s a -> BVar s b into the function a -> b that it represents, also computing its gradient a as well.

The Rank-N type forall s. Reifies s W => ... is used to ensure that BVars do not leak out of the context (similar to how it is used in Control.Monad.ST), and also as a reference to an ephemeral Wengert tape used to track the graph of references.

Turn a function BVar s a -> BVar s b into the function a -> b that it represents.

Benchmarks show that this should have virtually no overhead over directly writing a a -> b. BVar is, in this situation, a zero-cost abstraction, performance-wise.

See documentation of backprop for more information.

Take a function BVar s a -> BVar s b, interpreted as a function a -> b, and compute its gradient with respect to its input.

The resulting a -> a tells how the input (and its components) affects the output. Positive numbers indicate that the result will vary in the same direction as any adjustment in the input. Negative numbers indicate that the result will vary in the opposite direction as any adjustment in the input. Larger numbers indicate a greater sensitivity of change, and small numbers indicate lower sensitivity.

See documentation of backprop for more information.

If you want to provide an explicit "final gradient" for the end, see backpropWith.

A version of backprop that allows you to specify the gradent of your "final result" in with respect to the output of your function.

Typically, this is just the scalar 1, or a value of components that are all 1.

Instead of taking the b gradient, the you may provide a b -> b, which backpropWith calls with the result of your function as the argument. This allows you to return something with the correct "shape", if not a scalar.

backprop is essentially backpropWith with const 1 for scalars and Num instances.

Note that argument order changed in v0.2.4

Since: 0.2.0.0

backprop for a two-argument function.

Not strictly necessary, because you can always uncurry a function by passing in all of the argument inside a data type, or just use a tuple. However, this could potentially be more performant.

For 3 and more arguments, consider using backpropN.

evalBP for a two-argument function. See backprop2 for notes.

gradBP for a two-argument function. See backprop2 for notes.

backprop2, but allows you to provide the gradient of the "final result" with respect to the output of your function. See backpropWith for more details.

Note that argument order changed in v0.2.4

Since: 0.2.0.0

backprop generalized to multiple inputs of different types. See the Numeric.Backprop.Op for a mini-tutorial on heterogeneous lists.

Not strictly necessary, because you can always uncurry a function by passing in all of the inputs in a data type containing all of the arguments or a giant tuple. However, this could potentially also be more performant.

A Rec (BVar s) '[Double, Float, Double], for instance, is a tuple of BVar s Double, BVar s Float, and BVar s Double, and can be pattern matched on using :< (cons) and Ø (nil).

The RPureConstrained Backprop as in the constraint says that every value in the type-level list as must have a Backprop instance. This means you can use, say, '[Double, Float, Int], but not '[Double, Bool, String].

If you stick to concerete, monomorphic usage of this (with specific types, typed into source code, known at compile-time), then RPureConstrained Backprop as should be fulfilled automatically.

evalBP generalized to multiple inputs of different types. See documentation for backpropN for more details.

gradBP generalized to multiple inputs of different types. See documentation for backpropN for more details.

backpropN, but allows you to provide the gradient of the "final result" with respect to the output of your function. See backpropWith for more details.

Note that argument order changed in v0.2.4.

Since: 0.2.0.0

evalBP but with no arguments. Useful when everything is just given through constVar.

Lift a value into a BVar representing a constant value.

This value will not be considered an input, and its gradients will not be backpropagated.

Shorter alias for constVar, inspired by the ad library.

Since: 0.2.0.0

Coerce a BVar contents. Useful for things like newtype wrappers.

Since: 0.1.5.2

An infix version of viewVar, meant to evoke parallels to ^. from lens.

With normal values, you can extract something from that value with a lens:

x ^. myLens 

would extract a piece of x :: b, specified by myLens :: Lens' b a. The result has type a.

xVar ^^. myLens 

would extract a piece out of xVar :: BVar s b (a BVar holding a b), specified by myLens :: Lens' b a. The result has type BVar s a (a BVar holding a a)

This is the main way to pull out values from BVar of container types.

If you have control of your data type definitions, consider using splitBV, which lets you break out BVars of values into BVars of their individual fields automatically without requiring lenses.

NOTE: Usage of ^^. on many fields from the same item is usually the main source of overhead in backprop code, if you are looking to optimize your code. See <https://backprop.jle.im/07-performance.html this performance guide> for more information, and details on mitigating this overhead.

WARNING: Do not use with any lenses that operate "numerically" on the contents (like multiplying).

An infix version of setVar, meant to evoke parallels to .~ from lens.

With normal values, you can set something in a value with a lens:

x & myLens .~ y 

would "set" a part of x :: b, specified by myLens :: Lens' a b, to a new value y :: a.

xVar & myLens .~~ yVar 

would "set" a part of xVar :: BVar s b (a BVar holding a b), specified by myLens :: Lens' a b, to a new value given by yVar :: BVar s a. The result is a new (updated) value of type BVar s b.

This is the main way to set values inside BVars of container types.

Note that this does not incurr the performance overhead issues of viewVar and ^^., and is fairly cheap.

An infix version of overVar, meant to evoke parallels to %~ from lens.

With normal values, you can set modify in a value with a lens:

x & myLens %~ negate 

would "modify" a part of x :: b, specified by myLens :: Lens' a b, using the function negate :: a -> a.

xVar & myLens %~~ negate 

would "modify" a part of xVar :: BVar s b (a BVar holding a b), specified by myLens :: Lens' a b, using the function negate :: BVar s a -> BVar s . The result is a new (updated) value of type BVar s b.

Is essentially a convenient wrapper over a viewVar followed by a setVar.

Since: 0.2.4.0

An infix version of previewVar, meant to evoke parallels to ^? from lens.

With normal values, you can (potentially) extract something from that value with a lens:

x ^? myPrism 

would (potentially) extract a piece of x :: b, specified by myPrism :: Traversal' b a. The result has type Maybe a.

xVar ^^? myPrism 

would (potentially) extract a piece out of xVar :: BVar s b (a BVar holding a b), specified by myPrism :: Prism' b a. The result has type Maybe (BVar s a) (Maybe a BVar holding a a).

This is intended to be used with Prism's (which hits at most one target), but will actually work with any Traversal'. If the traversal hits more than one target, the first one found will be extracted.

This can be used to "pattern match" on BVars, by using prisms on constructors.

NOTE: Has the same potential of performance overhead issues as ^^.; see documentation of ^^. for more details.

An infix version of toListOfVar, meant to evoke parallels to ^.. from lens.

With normal values, you can extract all targets of a Traversal from that value with a:

x ^.. myTraversal 

would extract all targets inside of x :: b, specified by myTraversal :: Traversal' b a. The result has type [a].

xVar ^^.. myTraversal 

would extract all targets inside of xVar :: BVar s b (a BVar holding a b), specified by myTraversal :: Traversal' b a. The result has type [BVar s a] (A list of BVars holding as).

NOTE: Has all of the performance overhead issues of sequenceVar; see documentation for sequenceVar for more information.

An *UNSAFE* version of ^^? and previewVar assuming that the value is there.

Is undefined if the Traversal hits no targets.

Is essentially ^^? with fromJust, or ^^.. with head.

Since: 0.2.1.0

Using a Lens', extract a value inside a BVar. Meant to evoke parallels to view from lens.

See documentation for ^^. for more information, caveats, and warnings.

Using a Lens', set a value inside a BVar. Meant to evoke parallels to "set" from lens.

See documentation for .~~ for more information.

Using a Lens', modify a value inzide a BVar. Meant to evoke parallels to "over" from lens. See documentation for %~~ for more information.

Since: 0.2.4.0

Extract all of the BVars out of a Traversable container of BVars.

Note that this associates gradients in order of occurrence in the original data structure; the second item in the gradient is assumed to correspond with the second item in the input, etc.; this can cause unexpected behavior in Foldable instances that don't have a fixed number of items.

NOTE: A potential source of performance overhead. If there are \(n\) total elements, and you use \(m\) of them, then there is an overhead cost on the order of \(\mathcal{O}(m n)\), with a constant factor dependent on the cost of add. Should be negligible for types with cheap add (like Double), but may be costly for things like large matrices. See <https://backprop.jle.im/07-performance.html the performance guide> for for details.

Collect all of the BVars in a container into a BVar of that container's contents.

Note that this associates gradients in order of occurrence in the original data structure; the second item in the total derivative and gradient is assumed to correspond with the second item in the input, etc.; this can cause unexpected behavior in Foldable instances that don't have a fixed number of items.

Note that this does not suffer from the same performance overhead issues as sequenceVar. collectVar is \(\mathcal{O}(n)\), with a very small constant factor that consistent for all types. This reveals a general property of reverse-mode automatic differentiation; "many to one" is cheap, but "one to many" is expensive.

Using a Traversal', extract a single value inside a BVar, if it exists. If more than one traversal target exists, returns te first. Meant to evoke parallels to preview from lens. Really only intended to be used wth Prism's, or up-to-one target traversals.

See documentation for ^^? for more information, warnings, and caveats.

Using a Traversal', extract all targeted values inside a BVar. Meant to evoke parallels to toListOf from lens.

See documentation for ^^.. for more information, warnings, and caveats.

Useful pattern for constructing and deconstructing BVars of two-tuples.

Since: 0.2.1.0

Useful pattern for constructing and deconstructing BVars three-tuples.

Since: 0.2.1.0

Convert the value inside a BVar using a given isomorphism. Useful for things like constructors.

If you have control of your data type definitions, consider using joinBV, which lets you use your data type constructors themselves to join together BVars as their fields.

Warning: This is unsafe! It assumes that the isomorphisms themselves have derivative 1, so will break for things like exp & log. Basically, don't use this for any "numeric" isomorphisms.

Since: 0.1.4.0

Convert the values inside two BVars using a given isomorphism. Useful for things like constructors. See isoVar for caveats.

If you have control of your data type definitions, consider using joinBV, which lets you use your data type constructors themselves to join together BVars as their fields.

Since: 0.1.4.0

Convert the values inside three BVars using a given isomorphism. Useful for things like constructors. See isoVar for caveats.

Since: 0.1.4.0

Convert the values inside a tuple of BVars using a given isomorphism. Useful for things like constructors. See isoVar for caveats.

If you have control of your data type definitions, consider using joinBV, which lets you use your data type constructors themselves to join together BVars as their fields.

Since: 0.1.4.0

Lift an Op with an arbitrary number of inputs to a function on the appropriate number of BVars.

Should preferably be used only by libraries to provide primitive BVar functions for their types for users.

See Numeric.Backprop and documentation for liftOp for more information, and Numeric.Backprop.Op for a mini-tutorial on using Rec.

Lift an Op with a single input to be a function on a single BVar.

Should preferably be used only by libraries to provide primitive BVar functions for their types for users.

See Numeric.Backprop and documentation for liftOp for more information.

Lift an Op with two inputs to be a function on a two BVars.

Should preferably be used only by libraries to provide primitive BVar functions for their types for users.

See Numeric.Backprop and documentation for liftOp for more information.

Lift an Op with three inputs to be a function on a three BVars.

Should preferably be used only by libraries to provide primitive BVar functions for their types for users.

See Numeric.Backprop and documentation for liftOp for more information.

Split out a BVar of "higher-kinded data type", a la http://reasonablypolymorphic.com/blog/higher-kinded-data/

Lets you take BVar of a value into a separate BVar of every field of that value.

See Numeric.Backprop for a tutorial on usage.

This will work with all data types made with a single constructor, whose fields are all instances of Backprop, where the type itself has an instance of Backprop. The type also must derive Generic.

Note that access using splitBV and pattern matching is slightly slower than access using lenses (by about 10-20%).

See also BV, pattern synonym version where the deconstructor is exactly a view into splitBV.

NOTE: Like ^^. and viewVar, splitBV usage could potentially be the main source of performance overhead in your program. If your data type has \(n\) fields, and you use splitBV to later use \(m\) of those fields, there is an overhead cost on the order of \(\mathcal{O}(m n)\), with a constant factor dependent on the cost of add for your original data type. Should be negligible for types with cheap add (like Double), but may be costly for things like large matrices. See the performance guide for for details.

However, there is some potential opportunities to re-write some core library functionality that would allow splitBV to avoid all of the significant performance overhead issues of ^^.. Contact me if you are interested in helping out!

Since: 0.2.2.0

Assemble a BVar of "higher-kinded data type", a la http://reasonablypolymorphic.com/blog/higher-kinded-data/

It lets you take a BVar of every field of a value, and join them into a BVar of that value.

See Numeric.Backprop for a tutorial on usage.

This will work with all data types made with a single constructor, whose fields are all instances of Backprop, where the type itself has an instance of Backprop.

See also BV, a pattern synonym version where the constructor is exactly joinBV.

Note that joinBV does not suffer the major performance overhead issues of splitBV. This is a general property of reverse-mode automatic differentiation: "many to one" is cheap, but "one to many" is expensive.

Since: 0.2.2.0

Pattern synonym wrapping manual usage of splitBV and joinBV. It is a pattern for a BVar s (z f) containing a z (BVar s)

Since: 0.2.3.0

Helper class for generically "splitting" and "joining" BVars into constructors. See splitBV and joinBV.

See Numeric.Backprop for a tutorial on how to use this.

Instances should be available for types made with one constructor whose fields are all instances of Backprop, with a Generic instance.

Since: 0.2.2.0