Vector Norms and Orthogonality

Vector Spaces

A vector space is a set V of vectors on which two operations + and · are defined, called vector addition and scalar multiplication.

The operation + (vector addition) must satisfy the following conditions, where u, v, w are vectors and c,d are scalars:

1. Commutative law: u + v = v + u

2. Associative law: u + (v + w) = (u + v) + w

3. Additive identity: 0 + v = v and v + 0 = v

4. Additive inverses: u + (-u) = 0

The operation · (scalar multiplication) must satisfy the following conditions:

1. Distributive law over vector sum: c · (u + v) = c · u + c · v

2. Distributive law over scalar sum: (c+d) · v = c · v + d · v

3. Associative law of scalar product: c · (d · v) = (cd) · v

4. Unitary law: 1 · v = v

Vector Norm

Vector norm is a non-negative number which describes the length or extent of the vector in space. Also known as vector length or magnitude. There are different ways to calculate vector norms.

$$X =\begin{bmatrix}x_1 \\ x_2 \\ x_3 \end{bmatrix}$$

L1 Norm

L1 Norm is calculated as the sum of absolute values of the vector.

• Calculates the Manhattan distance from the origin of vector space

$$L^1 = |x_1| + |x_2|+|x_3|$$

L2 Norm

L2 Norm is calculated as the square root of the sum of the squared vector values.

• Calculates the Euclidean distance from the origin of vector space

• More commonly used in ML

$$L^2 = \sqrt{ x_1^2 + x_2^2+x_3^2}$$

Max Norm

Max Norm is calculated as the maximum vector values.

$$L^\infty = max(|x_1| , |x_2|,|x_3|)$$

Calculating Norm using NumPy

from numpy import array
from numpy.linalg import norm
from numpy import inf

x = array([1, 2, -3])
l1 = norm(x, 1)
l2 = norm(x, 2)
lmax = norm(x, inf)

print('L1 norm: ', l1)
print('L2 norm: ', l2)
print('LMax norm: ', lmax)

L1 norm:  6.0
L2 norm:  3.7416573867739413
LMax norm:  3.0

Orthogonality

Orthogonal: Two vectors are orthogonal if they are perpendicular to each other and their dot product is zero. u . v = 0

Proof: u · v = |u| |v| cos θ (From Cosine Similarity) Note: cosine of 90° = 0

Orthonormal: Two vectors are orthonormal if their dot product is zero and norm/length of each vector is 1. u . v = 0 and |u| = 1, |v| = 1

In 3-D Euclidean space, using L2 Norm to calculate length; we come up with the following equation for orthonormal vectors.

$$\sqrt{ u_1^2 + u_2^2+u_3^2} = 1 and \sqrt{ v_1^2 + v_2^2+v_3^2} = 1$$

Normal: A vector is said to be normal to a surface or curve if it is perpendicular to it.

Finding orthogonality using NumPy

from numpy import vdot
from numpy import array

u = array([2, 18])
v = array([3/2, -1/6])
dotProduct = vdot(u, v)
print(dotProduct)

0.0

Links: What is a Vector Space? (Abstract Algebra) Gentle Introduction to Vector Norms in Machine Learning From Norm to Orthogonality