Vectors and Vector Spaces

🚀 Vectors & Vector Spaces: The Foundation of Data

Linear Algebra is the study of linear mappings between vector spaces. In software engineering and machine learning, it provides the fundamental framework for representing data, model parameters, and geometric transformations.

🟢 Level 1: Core Definitions

1. Vector Spaces ( $\mathcal{V}$ )

A vector space is a set $V$ equipped with two operations: Vector Addition and Scalar Multiplication. To be a formal vector space, it must satisfy 8 axioms, including closure, commutativity, associativity, and the existence of identity and inverse elements.

Real Coordinate Space ( $\mathbb{R}^n$ ): The most common vector space in computing, consisting of all possible $n$ -tuples of real numbers.

2. Linear Combinations

A vector $\mathbf{v}$ is a linear combination of a set of vectors $\{\mathbf{v}_1, \dots, \mathbf{v}_n\}$ if it can be expressed as: $\mathbf{v} = c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \dots + c_n \mathbf{v}_n$ where $c_i$ are scalars.

import numpy as np

# Defining vectors in R^3
v1 = np.array([1, 2, 3])
v2 = np.array([4, 5, 6])

# A linear combination: 2*v1 + 0.5*v2
v_combined = 2 * v1 + 0.5 * v2
print(f"Linear Combination: {v_combined}")

🟡 Level 2: Basis and Dimension

3. Linear Independence

A set of vectors is linearly independent if no vector in the set can be written as a linear combination of the others. Formally: $\sum_{i=1}^n c_i \mathbf{v}_i = \mathbf{0} \implies c_1 = c_2 = \dots = c_n = 0$ If any $c_i \neq 0$ , the vectors are linearly dependent.

4. Span and Basis

Span: The set of all possible linear combinations of a set of vectors. It represents the “reach” of those vectors.
Basis: A linearly independent set of vectors that spans the entire space. It is the “minimal” set of vectors needed to reach every point in the space.
Dimension: The number of vectors in any basis for the space. For $\mathbb{R}^n$ , the dimension is $n$ .

5. Change of Basis

In Machine Learning (e.g., PCA), we often represent data in a new coordinate system that better captures the variance. We use a Transition Matrix $\mathbf{P}$ to move from basis $\mathcal{B}$ to basis $\mathcal{C}$ : $[\mathbf{x}]_{\mathcal{C}} = \mathbf{P}^{-1} [\mathbf{x}]_{\mathcal{B}}$

🔴 Level 3: Orthogonality and Norms

6. Dot Product and Similarity

The dot product measures the alignment between two vectors: $\mathbf{a} \cdot \mathbf{b} = \|\mathbf{a}\| \|\mathbf{b}\| \cos(\theta)$ .

Orthogonal: If $\mathbf{a} \cdot \mathbf{b} = 0$ , the vectors are at 90 degrees to each other.
Orthonormal: Orthogonal vectors that each have a magnitude (length) of 1.

7. Vector Norms ( $L_p$ Norms)

Norms measure the “size” or “length” of a vector, which is critical for regularization in ML.

$L_1$ Norm (Manhattan): $\sum |x_i|$ . Used in Lasso regression to promote sparsity.
$L_2$ Norm (Euclidean): $\sqrt{\sum x_i^2}$ . The standard distance metric; used in Ridge regression to prevent large outliers.
$L_{\infty}$ Norm (Max): $\max |x_i|$ . The maximum absolute component.

# Calculating norms with NumPy
x = np.array([3, -4])

l1_norm = np.linalg.norm(x, ord=1)  # 3 + 4 = 7
l2_norm = np.linalg.norm(x, ord=2)  # sqrt(3^2 + (-4)^2) = 5
print(f"L1: {l1_norm}, L2: {l2_norm}")