Vector(Matrix) Derivatives

Matrix/vector manipulation

You should be comfortable with these rules. They will come in handy when you want to simplify an expression before differentiating

• $(AB)^T = B^TA^T$
• $(a^TBc)^T = c^TB^Ta$
• $a^Tb = b^Ta$ (the result is a scalar, and the transpose of a scalar is itself)
• $(A + B)C = AC + BC$ (multiplication is distributive)
• $(a + b)^T C = a^TC + b^T C$
• $AB {=}\mathllap{/\,} BA$ (multiplication is not commutative)

Common Vector Derivatives

Scalar derivative $f(x) \to \frac{df}{dx}$	Vector derivative $f(X) \to \frac{df}{dX}$
bx → b	$x^TB \to B$
bx → b	$x^Tb \to b$
x2 → 2x	$x^Tx \to 2x$
$bx^2 \to 2bx$	$x^TBx \to 2Bx$

Common Derivatives Rule

• $\frac{dg(u)}{x} = \frac{dg(u)}{u}\frac{du}{dx}$

• $\frac{df(g(u))}{dx} = \frac{d(f(g))}{dg}\frac{dg(u)}{du}\frac{du}{dx}$

  • $u \to g \to f$, all vector

• $X \to Y = AX + B \to l = f(Y)$, $\frac{dl}{dX} = A^T\frac{dl}{dY}$ (“chain rule” for linear function)

  • $X \to Y \to l$, l scalar, X, Y vector/matrix
  • proof, let $\frac{dl}{dX}$ is a m x 1 vector, $\frac{dl}{dY}$ is a n x 1 vector and $\frac{dY}{dX}$ is a n x m vector

  $$\frac{dl}{dX} = \frac{dl}{dY}\frac{dY}{dX}$$

  $$(\frac{dl}{dX})^T = (\frac{dY}{dX})^T(\frac{dl}{dY})^T$$

  $$\frac{dl}{dX} = (\frac{dY}{dX})^T\frac{dl}{dY}$$

• $X \to Y = XA + B \to l = f(Y)$, $\frac{dl}{dX} = \frac{dl}{dY}A^T$

  • $X \to Y \to l$, l scalar, X, Y vector/matrix

Common Differentiate Rule

• $d(X^T) = d(X)^T$
• $d(X + Y) = d(X) + d(Y)$
• $d(XY) = (dX)Y + X(dY)$

From Differentiate to Derivatives

• $df = (\frac{df}{dx})^Tdx$

Derivatives of Matrices, Vectors and Scalar Forms

First-order(Linear Products)

Let $f = x^Ta$, compute $\frac{df}{dx}$

Solution

$$df = d(x^T)a = d(x)^Ta = a^Tdx \to \frac{df}{dx} = a$$

We also have $\frac{d(a^Tx)}{dx} = a$

---

Let $f = a^TXb$, compute $\frac{df}{dX}$

Solution

using $AB = BA$ when AB is a scalar,

$$df = a^Td{X}b = ba^TdX \to \frac{df}{dX} = ab^T$$

---

Let $f = a^TX^Tb$, compute $\frac{df}{dX}$

Solution

$$df = a^Td(X^T)b = a^Td(X)^Tb = (d(X)a)^Tb = b^Td(X)a = ab^Td(X) \to \frac{df}{dX} = ba^T$$

Second-order(Quadratic Products)

Let $f = b^TX^TXc$, compute $\frac{df}{dX}$

Solution

$$\tag{1} df = b^Td(X^T)Xc + b^TX^Td(X)c$$

From (1)

$$\tag{2} b^Td(X^T)Xc = (Xc)^Td(X)b = b(Xc)^Td(X) = bc^TX^Td(X) = (Xcb^T)^Td(X)$$

$$\tag{3} b^TX^Td(X)c = cb^TX^Td(X) = (Xbc^T)^Td(X)$$

Replace (2),(3) into (1)

$$df = ((Xcb)^T + (Xbc^T)^T)d(X) \to \frac{df}{dX} = X(cb^T + bc^T)$$

---

Let $f = (X\theta - y)^T(X\theta - y)$, compute $\frac{df}{d\theta}$

Solution 1

$$\tag{1} df = d((X\theta - y)^T)(X\theta - y) + (X\theta - y)^Td(X\theta - y)$$

From (1), we need to compute $d((X\theta - y)^T)$ and $d(X\theta - y)$

$$\tag{2} \begin{aligned} d((X\theta - y)^T) &= (d(X\theta - y))^T\\ &= (Xd\theta)^T \end{aligned}$$

$$\tag{3} \begin{aligned} d(X\theta - y) &= Xd\theta \end{aligned}$$

Apply (2)(3) into (1),

$$\begin{aligned} df &= (Xd\theta)^T(X\theta - y) + (X\theta - y)^TXd\theta \\ &= (X\theta - y)^TXd\theta + (X\theta - y)^TXd\theta \\ &= 2(X\theta - y)^TXd(X) \end{aligned}$$

From $df = (\frac{df}{dx})^Tdx$,

$$\frac{df}{d\theta} = 2X^T(X\theta - y)$$

Solution 2

$$\tag{1} \frac{df}{d\theta} = \frac{df}{d(X\theta - y)} \frac{d(X\theta - y)}{d\theta}$$

Let $u = X\theta - y$, we need to compute $\frac{du^Tu}{du}$ and $\frac{d(X\theta - y)}{d\theta}$

$$\begin{aligned} df &= du^Tu + u^Tdu\\ &= (du)^Tu + u^Tdu\\ &= u^Tdu+u^Tdu\\ &= 2u^Tdu \end{aligned}$$

$$\tag{2} \frac{df}{du} = 2u$$

$$d(X\theta - y) = Xd\theta$$

Considering, $dz = \frac{dz}{dx} dx$, when $z$ and $x$ are vectors

$$\tag{3} \frac{d(X\theta - y)}{d\theta} = X$$

hence, apply (2) to (1),

$$\frac{df}{d\theta} = 2X^T(X\theta - y)$$

---

$J(\theta) = ylog(sigmod(\theta^TX)) + (1 - y)log(1 - sigmod(\theta^TX))$, compute $\frac{dJ(\theta)}{\theta}$
Solution
Let $u = sigmond(\theta^TX), v = \theta^TX$,

$$\frac{dJ}{d\theta} = \frac{ylog(u) + (1 - y)log(1 - u)}{du}\frac{du}{dv}\frac{dv}{\theta}$$