Linear Algebra

Linear Independence

Imron Rosyadi

Introduction to Linear Independence

In applications, knowing if vectors are interrelated is crucial. An unecessary vector can complicate analysis.

Consider \(\mathbf{i} = (1,0)\), \(\mathbf{j} = (0,1)\) in \(R^2\).

Any vector \((x,y)\) is uniquely \(x\mathbf{i} + y\mathbf{j}\).

Example: \((3,2) = 3\mathbf{i} + 2\mathbf{j}\).

Note

This is the only way to express \((3,2)\) using \(\mathbf{i}\) and \(\mathbf{j}\).

Now, add \(\mathbf{w} = \left(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}\right)\) (unit vector at \(45^\circ\)).

\((3,2)\) can be expressed in infinitely many ways:

• \(3\mathbf{i} + 2\mathbf{j} + 0\mathbf{w}\)
• \(2\mathbf{i} + \mathbf{j} + \sqrt{2}\mathbf{w}\)
• \(4\mathbf{i} + 3\mathbf{j} - \sqrt{2}\mathbf{w}\)

The vector \(\mathbf{w}\) is unnecessary because: \(\mathbf{w} = \frac{1}{\sqrt{2}} \mathbf{i} + \frac{1}{\sqrt{2}} \mathbf{j}\)

Start by introducing the core idea of linear independence: whether a set of vectors contains redundant information. Use the simple 2D coordinate system as an analogy. Emphasize how adding a vector that can be formed by others creates ambiguity in representation, which is generally undesirable in systems. The “superfluous” concept is key here.

Defining Linear Independence

DEFINITION 1:

A set \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{r}\}\) of two or more vectors in a vector space \(V\) is linearly independent if no vector in \(S\) can be expressed as a linear combination of the others.

A set that is not linearly independent is linearly dependent.

For a single vector set \(\{\mathbf{v}\}\), it is linearly independent if and only if \(\mathbf{v} \neq \mathbf{0}\).

Important

Key Theorem 4.3.1: A nonempty set \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{r}\}\) in a vector space \(V\) is linearly independent if and only if the only coefficients satisfying the vector equation: \[ k_{1} \mathbf{v}_{1} + k_{2} \mathbf{v}_{2} + \dots + k_{r} \mathbf{v}_{r} = \mathbf{0} \] are \(k_{1} = 0, k_{2} = 0, \ldots , k_{r} = 0\).

This slide formally defines linear independence and dependence. The most crucial part is Theorem 4.3.1, which provides a practical method for checking independence. Emphasize that finding any non-zero coefficients means the set is dependent. This theorem transforms the problem into solving a homogeneous linear system.

Example: Standard Unit Vectors in \(R^n\)

The standard unit vectors \(\mathbf{e}_{1}, \mathbf{e}_{2}, \ldots , \mathbf{e}_{n}\) are linearly independent.

For \(R^3\): \(\mathbf{i} = (1,0,0)\), \(\mathbf{j} = (0,1,0)\), \(\mathbf{k} = (0,0,1)\)

Consider the equation: \(k_{1}\mathbf{i} + k_{2}\mathbf{j} + k_{3}\mathbf{k} = \mathbf{0}\)

In component form: \(k_{1}(1,0,0) + k_{2}(0,1,0) + k_{3}(0,0,1) = (0,0,0)\) \((k_{1}, k_{2}, k_{3}) = (0, 0, 0)\)

This directly implies \(k_{1} = 0, k_{2} = 0, k_{3} = 0\). Thus, they are linearly independent.

Let’s verify this in Python.

This example applies Theorem 4.3.1 to a familiar set of vectors. The Python code reinforces the idea that if the determinant of the matrix formed by the vectors is non-zero, the homogeneous system has only the trivial solution, confirming linear independence. This is a common method in ECE, especially when dealing with system matrices.

Example: Linear Independence in \(R^3\)

Determine if \(\mathbf{v}_{1} = (1, -2, 3)\), \(\mathbf{v}_{2} = (5, 6, -1)\), \(\mathbf{v}_{3} = (3, 2, 1)\) are linearly independent.

We set up the equation: \(k_{1} \mathbf{v}_{1} + k_{2} \mathbf{v}_{2} + k_{3} \mathbf{v}_{3} = \mathbf{0}\)

This translates to the homogeneous system: \[ \begin{array}{r} k_{1} + 5k_{2} + 3k_{3} = 0 \\ -2k_{1} + 6k_{2} + 2k_{3} = 0 \\ 3k_{1} - k_{2} + k_{3} = 0 \end{array} \]

The coefficient matrix is: \[ A = \left[ \begin{array}{ccc} 1 & 5 & 3 \\ -2 & 6 & 2 \\ 3 & -1 & 1 \end{array} \right] \] We need to check if this system has non-trivial solutions.

Tip

For a square matrix, if \(\operatorname{det}(A) = 0\), then \(A\mathbf{x} = \mathbf{0}\) has non-trivial solutions, implying linear dependence. If \(\operatorname{det}(A) \neq 0\), only the trivial solution exists, implying linear independence.

This slide sets up the problem for a more complex R^3 example. It explicitly shows how the vector equation translates into a system of linear equations. The callout highlights the determinant test, a powerful tool for square matrices.

Example: \(R^3\) (Cont.) - Python Solution

Let’s use Python to solve the system and find the determinant.

This interactive slide directly solves the system from the previous slide using numpy. It calculates the determinant and then, because it’s zero, demonstrates a non-trivial solution, confirming linear dependence. This is a practical application of computational tools in linear algebra.

Example: Linear Independence in \(R^4\)

Determine if \(\mathbf{v}_{1} = (1,2,2, -1)\), \(\mathbf{v}_{2} = (4,9,9, -4)\), \(\mathbf{v}_{3} = (5,8,9, -5)\) are linearly independent in \(R^4\).

We need to solve \(k_{1}\mathbf{v}_{1} + k_{2}\mathbf{v}_{2} + k_{3}\mathbf{v}_{3} = \mathbf{0}\). This leads to the system: \[ \begin{array}{r} k_{1} + 4k_{2} + 5k_{3} = 0 \\ 2k_{1} + 9k_{2} + 8k_{3} = 0 \\ 2k_{1} + 9k_{2} + 9k_{3} = 0 \\ -k_{1} - 4k_{2} - 5k_{3} = 0 \end{array} \]

This example demonstrates a case of linear independence for a non-square system. It introduces the concept of checking the rank of the matrix. If the rank equals the number of columns (variables), it implies only the trivial solution exists, thus linear independence. This is a common technique in numerical linear algebra.

Linear Independence of Polynomials

The set of polynomials \(\{1, x, x^2, \ldots, x^n\}\) forms a linearly independent set in \(P_n\).

Consider the equation: \(a_{0}(1) + a_{1}(x) + a_{2}(x^{2}) + \dots +a_{n}(x^{n}) = \mathbf{0}\)

This means \(a_{0} + a_{1}x + a_{2}x^{2} + \dots +a_{n}x^{n} = 0\) for all \(x \in (-\infty, \infty)\).

Tip

A non-zero polynomial of degree \(n\) has at most \(n\) distinct roots. If a polynomial is zero for all \(x\), it must be the zero polynomial, meaning all its coefficients are zero. Therefore, \(a_{0} = a_{1} = a_{2} = \dots = a_{n} = 0\).

This confirms the linear independence of the set \(\{1, x, x^2, \ldots, x^n\}\).

This slide covers the linear independence of the standard basis for polynomial spaces. The key insight here is the property of polynomials regarding their roots. If a polynomial equation holds for all x, then all its coefficients must be zero. This is fundamental for understanding polynomial spaces.

Example: Linear Independence of Specific Polynomials

Determine if \(\mathbf{p}_{1} = 1 - x\), \(\mathbf{p}_{2} = 5 + 3x - 2x^{2}\), \(\mathbf{p}_{3} = 1 + 3x - x^{2}\) are linearly independent in \(P_2\).

Set up the equation:

\(k_{1}\mathbf{p}_{1} + k_{2}\mathbf{p}_{2} + k_{3}\mathbf{p}_{3} = \mathbf{0}\)

Substitute the polynomials:

\(k_{1}(1 - x) + k_{2}(5 + 3x - 2x^{2}) + k_{3}(1 + 3x - x^{2}) = 0\)

Group terms by powers of \(x\):

\((k_{1} + 5k_{2} + k_{3}) + (-k_{1} + 3k_{2} + 3k_{3})x + (-2k_{2} - k_{3})x^{2} = 0\)

For this to be true for all \(x\), each coefficient must be zero: \[ \begin{array}{c} k_{1}+5k_{2}+ k_{3}=0 \\ -k_{1}+3k_{2}+3k_{3}=0 \\ -2k_{2}- k_{3}=0 \end{array} \]

Let’s solve this system using Python.

This example shows how to reduce the problem of polynomial linear independence to solving a system of linear equations. The interactive Python code confirms the determinant is zero, thus leading to linear dependence. This is a practical method for checking independence in function spaces using matrix methods.

Special Cases and Geometric Interpretation

THEOREM 4.3.2:

1. A finite set that contains \(\mathbf{0}\) is linearly dependent.

2. A set with exactly one vector \(\{\mathbf{v}\}\) is linearly independent iff \(\mathbf{v} \neq \mathbf{0}\).

3. A set with exactly two vectors \(\{\mathbf{v}_1, \mathbf{v}_2\}\) is linearly independent iff neither vector is a scalar multiple of the other.

Note

Proof of (a):

For any vectors \(\mathbf{v}_{1},\ldots ,\mathbf{v}_{r}\), the set \(S = \{\mathbf{v}_{1},\ldots ,\mathbf{v}_{r},\mathbf{0}\}\) is linearly dependent since \(0\mathbf{v}_{1} + \dots +0\mathbf{v}_{r} + 1(\mathbf{0}) = \mathbf{0}\) uses non-zero coefficient \(1\) for \(\mathbf{0}\).

Special Cases and Geometric Interpretation (Cont.)

Geometric Interpretation:

• Two vectors in \(R^2\) or \(R^3\) are linearly independent if and only if they do not lie on the same line (when originating from the origin).

  Otherwise, one is a scalar multiple of the other.

1. Linearly dependent

3. Linearly independent

This slide consolidates simple cases and offers valuable geometric intuition. For ECE students, visualizing these concepts can greatly aid understanding. Emphasize that linear dependence implies collinearity for two vectors.

Geometric Interpretation (Cont.)

• Three vectors in \(R^3\) are linearly independent if and only if they do not lie in the same plane (when originating from the origin). Otherwise, at least one would be a linear combination of the other two.

2. Linearly dependent

3. Linearly independent

Continuing the geometric interpretation, this slide illustrates linear dependence for three vectors in \(R^3\). If they lie in the same plane, one can be expressed as a combination of the others. This is a crucial visual for understanding the concept in 3D space.

Maximum Number of Linearly Independent Vectors

THEOREM 4.3.3: Let \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{r}\}\) be a set of vectors in \(R^{n}\). If \(r > n\), then \(S\) is linearly dependent.

Important

This means in \(R^n\), you can have at most \(n\) linearly independent vectors.

• In \(R^2\), at most 2 linearly independent vectors.
• In \(R^3\), at most 3 linearly independent vectors.

Proof Sketch:

1. Form the homogeneous system \(k_{1}\mathbf{v}_{1} + \dots + k_{r}\mathbf{v}_{r} = \mathbf{0}\).
2. This system has \(n\) equations and \(r\) unknowns (\(k_1, \ldots, k_r\)).
3. If \(r > n\) (more unknowns than equations), the system must have non-trivial solutions.
4. Therefore, \(S\) is linearly dependent.

Theorem 4.3.3 is fundamental for understanding the dimensionality of vector spaces. Explain that having more vectors than the dimension of the space guarantees redundancy. Connect this to real-world ECE scenarios like sensor networks: if you have more sensors than the degrees of freedom in your system, some sensor readings must be linearly dependent on others. This implies redundancy and potentially unnecessary complexity.

Linear Independence of Functions: The Wronskian

Determining linear independence for functions can be complex.

DEFINITION 2: The Wronskian

If \(\mathbf{f}_{1}(x),\ldots ,\mathbf{f}_{n}(x)\) are functions that are \(n-1\) times differentiable on an interval, then the determinant: \[ W(x) = \begin{vmatrix} f_{1}(x) & f_{2}(x) & \dots & f_{n}(x) \\ f_{1}^{\prime}(x) & f_{2}^{\prime}(x) & \dots & f_{n}^{\prime}(x) \\ \vdots & \vdots & & \vdots \\ f_{1}^{(n - 1)}(x) & f_{2}^{(n - 1)}(x) & \dots & f_{n}^{(n - 1)}(x) \end{vmatrix} \] is called the Wronskian of \(f_{1},f_{2},\ldots ,f_{n}\).

THEOREM 4.3.4:

If \(\mathbf{f}_{1},\ldots ,\mathbf{f}_{n}\) have \(n-1\) continuous derivatives on \((-\infty, \infty)\), and if \(W(x)\) is not identically zero on \((-\infty, \infty)\), then these functions form a linearly independent set.

Warning

WARNING: If \(W(x)\) is identically zero, no conclusion can be reached about linear independence. The functions may be linearly independent or dependent.

Introduce the Wronskian as a tool, particularly relevant in differential equations and control theory (ECE applications). Explain its definition and the key theorem. Crucially, highlight the warning about the converse not being true, as this is a common point of confusion.

Example: Wronskian for \(x\) and \(\sin x\)

Show that \(\mathbf{f}_{1} = x\) and \(\mathbf{f}_{2} = \sin x\) are linearly independent using the Wronskian.

Here, \(n=2\). We need \(n-1=1\) derivative.

\(f_1(x) = x\), \(f_1'(x) = 1\)

\(f_2(x) = \sin x\), \(f_2'(x) = \cos x\)

The Wronskian is: \[ W(x) = \begin{vmatrix} x & \sin x \\ 1 & \cos x \end{vmatrix} = x \cos x - \sin x \]

This function is not identically zero. For example, \(W\left(\frac{\pi}{2}\right) = \frac{\pi}{2} \cos \left(\frac{\pi}{2}\right) - \sin \left(\frac{\pi}{2}\right) = \frac{\pi}{2}(0) - 1 = -1\).

Since \(W(x)\) is not identically zero, \(\mathbf{f}_{1}\) and \(\mathbf{f}_{2}\) are linearly independent.

This slide provides a concrete example of calculating and interpreting the Wronskian. The calculation is straightforward, and the conclusion is drawn based on Theorem 4.3.4.

Interactive Wronskian Plot: \(x \cos x - \sin x\)

Let’s visualize the Wronskian \(W(x) = x \cos x - \sin x\).

This interactive plot visually demonstrates that the Wronskian \(W(x) = x \cos x - \sin x\) is not identically zero. Students can see that the function has many non-zero values, reinforcing the concept of linear independence. This is a great way to engage ECE students who appreciate visual data.

Example: Wronskian for \(1, e^x, e^{2x}\)

Show that \(\mathbf{f}_{1} = 1\), \(\mathbf{f}_{2} = e^{x}\), and \(\mathbf{f}_{3} = e^{2x}\) are linearly independent.

Here, \(n=3\). We need \(n-1=2\) derivatives.

\(f_1(x) = 1\), \(f_1'(x) = 0\), \(f_1''(x) = 0\)

\(f_2(x) = e^x\), \(f_2'(x) = e^x\), \(f_2''(x) = e^x\)

\(f_3(x) = e^{2x}\), \(f_3'(x) = 2e^{2x}\), \(f_3''(x) = 4e^{2x}\)

The Wronskian is: \[ W(x) = \begin{vmatrix} 1 & e^{x} & e^{2x} \\ 0 & e^{x} & 2e^{2x} \\ 0 & e^{x} & 4e^{2x} \end{vmatrix} \] Expand along the first column:

\(W(x) = 1 \cdot \begin{vmatrix} e^{x} & 2e^{2x} \\ e^{x} & 4e^{2x} \end{vmatrix} - 0 + 0\)

\(W(x) = e^{x}(4e^{2x}) - 2e^{2x}(e^{x}) = 4e^{3x} - 2e^{3x} = 2e^{3x}\)

Since \(W(x) = 2e^{3x}\) is never zero for any \(x\), these functions are linearly independent.

Another Wronskian example, this time with exponential functions. The calculation is more involved but still manageable. Emphasize that \(2e^{3x}\) is always positive and thus never zero, which is the key to concluding linear independence.

Interactive Wronskian Plot: \(2e^{3x}\)

Let’s visualize the Wronskian \(W(x) = 2e^{3x}\).

This interactive plot for \(W(x) = 2e^{3x}\) clearly shows that the Wronskian is always positive and never crosses the x-axis, thus confirming it is not identically zero. This further strengthens the understanding of the Wronskian criterion for linear independence.

Proof of Theorem 4.3.1 (Outline)

Theorem 4.3.1: \(S = \{\mathbf{v}_{1}, \ldots , \mathbf{v}_{r}\}\) is linearly independent \(\iff\) the only solution to \(k_{1} \mathbf{v}_{1} + \dots + k_{r} \mathbf{v}_{r} = \mathbf{0}\) is \(k_1 = \dots = k_r = 0\).

Part 1: \((\implies)\) If \(S\) is linearly independent, then \(k_1 = \dots = k_r = 0\).

1. Assume \(S\) is linearly independent.
2. Suppose there is a solution with at least one \(k_i \neq 0\).
3. Without loss of generality, let \(k_1 \neq 0\).
4. Then \(\mathbf{v}_{1} = \left(-\frac{k_{2}}{k_{1}}\right)\mathbf{v}_{2} + \dots + \left(-\frac{k_{r}}{k_{1}}\right)\mathbf{v}_{r}\).
5. This expresses \(\mathbf{v}_1\) as a linear combination of others, contradicting linear independence.
6. Therefore, all \(k_i\) must be zero.

Proof of Theorem 4.3.1 (Outline)

Part 2: \((\impliedby)\) If the only solution is \(k_1 = \dots = k_r = 0\), then \(S\) is linearly independent.

1. Assume the only solution to \(k_{1} \mathbf{v}_{1} + \dots + k_{r} \mathbf{v}_{r} = \mathbf{0}\) is \(k_1 = \dots = k_r = 0\).
2. Suppose \(S\) is linearly dependent.
3. By definition, some vector (say \(\mathbf{v}_1\)) can be written as \(\mathbf{v}_{1} = c_{2}\mathbf{v}_{2} + \dots + c_{r}\mathbf{v}_{r}\).
4. Rearranging gives \(\mathbf{v}_{1} + (-c_{2})\mathbf{v}_{2} + \dots + (-c_{r})\mathbf{v}_{r} = \mathbf{0}\).
5. This is a non-trivial solution (coefficient of \(\mathbf{v}_1\) is \(1 \neq 0\)).
6. This contradicts our initial assumption.
7. Therefore, \(S\) must be linearly independent.

This slide provides a concise outline of the proof for Theorem 4.3.1. While the full proof might not be required for every ECE student, understanding the logic behind it is important. Emphasize the proof by contradiction approach for both directions.