Matrix Operations and Linear Transformations

🚀 Matrix Operations: Transforming Data

A matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$ is a 2D array of scalars. Beyond being a simple data structure, a matrix represents a Linear Transformation from an $n$ -dimensional space to an $m$ -dimensional space.

🟢 Level 1: Core Operations

1. Matrix Multiplication ( $\mathbf{C} = \mathbf{A}\mathbf{B}$ )

Matrix multiplication is not element-wise. Instead, the entry $C_{ij}$ is the dot product of the $i$ -th row of $\mathbf{A}$ and the $j$ -th column of $\mathbf{B}$ : $C_{ij} = \sum_{k=1}^n A_{ik} B_{kj}$

import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Matrix Multiplication (Dot Product)
C = np.dot(A, B)  # or A @ B
print(f"Matrix Product:\n{C}")

# Element-wise (Hadamard) Product
C_element = A * B
print(f"Element-wise Product:\n{C_element}")

2. Transpose and Special Matrices

Transpose ( $\mathbf{A}^T$ ): Formed by swapping rows and columns ( $A_{ij} \to A_{ji}$ ).
Symmetric Matrix: A square matrix where $\mathbf{A} = \mathbf{A}^T$ .
Identity Matrix ( $\mathbf{I}$ ): A square matrix with 1s on the diagonal and 0s elsewhere. $\mathbf{A}\mathbf{I} = \mathbf{A}$ .
Orthogonal Matrix: A square matrix where $\mathbf{A}^T\mathbf{A} = \mathbf{I}$ . Its columns are orthonormal.

🟡 Level 2: Rank and Invertibility

3. Matrix Rank

The rank of a matrix is the number of linearly independent rows or columns. It represents the dimension of the output space after the transformation.

Full Rank: A matrix has full rank if its rank equals the smaller of its dimensions.

4. Determinant ( $\det(\mathbf{A})$ )

The determinant is a scalar value that represents the “volume scaling factor” of the linear transformation.

If $\det(\mathbf{A}) = 0$ , the transformation collapses the space into a lower dimension.
If $\det(\mathbf{A}) \neq 0$ , the matrix is invertible.

5. Matrix Inverse ( $\mathbf{A}^{-1}$ )

For a square, non-singular matrix, the inverse satisfies $\mathbf{A}\mathbf{A}^{-1} = \mathbf{I}$ . It “undoes” the transformation performed by $\mathbf{A}$ .

# Calculating Determinant and Inverse
A = np.array([[1, 2], [3, 4]])

det_A = np.linalg.det(A)
inv_A = np.linalg.inv(A)

print(f"Determinant: {det_A}")
print(f"Inverse:\n{inv_A}")

🔴 Level 3: Advanced Concepts

6. The Moore-Penrose Pseudoinverse ( $\mathbf{A}^+$ )

When a matrix is not square or is singular, we use the pseudoinverse for solving linear systems: $\mathbf{A}^+ = (\mathbf{A}^T \mathbf{A})^{-1} \mathbf{A}^T$ ML Application: This is the mathematical foundation for finding the optimal weights in Ordinary Least Squares (OLS) linear regression.

7. Matrix Trace

The trace is the sum of the diagonal elements of a square matrix: $\text{tr}(\mathbf{A}) = \sum A_{ii}$ . It is invariant under cyclic permutations and basis changes.