Linear Algebra

4.7 Row Space, Column Space, and Null Space

Imron Rosyadi

Why These Spaces Matter in ECE?

• Analyze Circuit Behavior:
  • Determine if a desired voltage/current pattern (output) is achievable given component constraints (matrix columns).
• Design Control Systems:
  • Understand the ‘reachable’ states of a system (column space) and the states that don’t affect the output (null space).
• Process Signals & Images:
  • Identify redundant information (null space) or extract essential features (basis for column/row space).
• Solve Optimization Problems:
  • Find optimal solutions within the constraints defined by these spaces.

Note

Think of a linear system as a black box:

• Inputs \(\rightarrow\) System Matrix \(\rightarrow\) Outputs
• The “spaces” help us peek inside this box and understand its capabilities and limitations.

Good morning/afternoon everyone! Today we’re diving into a crucial topic in linear algebra: row space, column space, and null space. For ECE students, these aren’t just abstract mathematical concepts; they are powerful tools that help us understand and design the systems we work with.

Imagine you’re designing a filter for an audio signal or a control algorithm for a robot. Linear algebra, and specifically these vector spaces, provide the framework to analyze how your system behaves.

For instance, the column space can tell us if a desired output, like a specific voltage waveform, is even possible to generate with our circuit components. The null space helps us understand what inputs might produce zero output, or what internal states of a system are ‘hidden’ from our sensors. The row space relates to the input-output behavior.

These concepts are fundamental to areas like signal processing, control theory, circuit analysis, and even machine learning.

Revisiting Matrices: Rows and Columns

We’re all familiar with matrices. Let’s formally define their row and column vectors.

For an \(m \times n\) matrix \(A\):

\[ A = \left[ \begin{array}{cccc}a_{11} & a_{12} & \dots & a_{1n} \\ a_{21} & a_{22} & \dots & a_{2n} \\ \vdots & \vdots & & \vdots \\ a_{m1} & a_{m2} & \dots & a_{mn} \end{array} \right] \]

Row Vectors of \(A\) (in \(R^n\)):

\[ \begin{array}{l}{{\bf r}_{1}=[a_{11}\quad a_{12}\quad\cdots\quad a_{1n}]}\\ {{\bf r}_{2}=[a_{21}\quad a_{22}\quad\cdots\quad a_{2n}]}\\ {\vdots\qquad\vdots}\\ {{\bf r}_{m}=[a_{m1}\quad a_{m2}\quad\cdots\quad a_{mn}]}\end{array} \]

Column Vectors of \(A\) (in \(R^m\)):

\[ \mathbf{c}_{1} = \left[ \begin{array}{c}a_{11} \\ a_{21} \\ \vdots \\ a_{m1} \end{array} \right], \quad \mathbf{c}_{2} = \left[ \begin{array}{c}a_{12} \\ a_{22} \\ \vdots \\ a_{m2} \end{array} \right], \ldots , \quad \mathbf{c}_{n} = \left[ \begin{array}{c}a_{1n} \\ a_{2n} \\ \vdots \\ a_{mn} \end{array} \right] \]

Revisiting Matrices: Rows and Columns

EXAMPLE 1: Row and Column Vectors of a \(2 \times 3\) Matrix

Let \(A = \left[ \begin{array}{rrr}2 & 1 & 0 \\ 3 & -1 & 4 \end{array} \right]\).

• Row Vectors: \(\mathbf{r}_{1} = [2 \quad 1 \quad 0]\) and \(\mathbf{r}_{2} = [3 \quad -1 \quad 4]\)
• Column Vectors: \(\mathbf{c}_{1} = \left[ \begin{array}{c}2 \\ 3 \end{array} \right], \quad \mathbf{c}_{2} = \left[ \begin{array}{c}1 \\ -1 \end{array} \right], \quad \text{and} \quad \mathbf{c}_{3} = \left[ \begin{array}{c}0 \\ 4 \end{array} \right]\)

We’ll start with the basics. A matrix is just a rectangular array of numbers. But we can view these numbers in two important ways: as rows or as columns. Each row forms a row vector, and each column forms a column vector.

It’s important to note the dimensions: if \(A\) is \(m \times n\), its row vectors are in \(R^n\) (they have \(n\) components), and its column vectors are in \(R^m\) (they have \(m\) components). This distinction will be crucial when we define the vector spaces.

Example 1 simply illustrates this. For a 2x3 matrix, we have two row vectors, each with 3 components, and three column vectors, each with 2 components.

Defining the Core Spaces

These three vector spaces are fundamental to understanding matrix properties and linear systems.

Important

DEFINITION 2: Row Space, Column Space, and Null Space

If \(A\) is an \(m \times n\) matrix:

1. Row Space of \(A\) (\(\text{row}(A)\)):
   • The subspace of \(R^{n}\) spanned by the row vectors of \(A\).
   • It represents all possible linear combinations of the row vectors.
2. Column Space of \(A\) (\(\text{col}(A)\)):
   • The subspace of \(R^{m}\) spanned by the column vectors of \(A\).
   • It represents all possible linear combinations of the column vectors.
3. Null Space of \(A\) (\(\text{null}(A)\)):
   • The solution space of the homogeneous system \(A\mathbf{x} = \mathbf{0}\).
   • It is a subspace of \(R^{n}\).

Note

ECE Perspective:

• Column Space: All possible output vectors (signals, states) that the system can produce from its inputs.
• Row Space: All possible input patterns that can be “seen” by the system’s output.
• Null Space: The set of inputs that produce a zero output. These inputs are “invisible” or “neutralized” by the system.

Now we get to the heart of the matter: defining these three key vector spaces.

The row space is simply all the vectors you can create by taking linear combinations of the matrix’s row vectors. It lives in \(R^n\).

The column space is all the vectors you can create by taking linear combinations of the matrix’s column vectors. It lives in \(R^m\). This space is particularly important for understanding what outputs a system can achieve.

The null space is a bit different. It’s the set of all vectors \(\mathbf{x}\) such that when you multiply them by matrix \(A\), you get the zero vector. In simpler terms, these are the inputs that the system “annihilates” or “doesn’t respond to” in terms of output. It lives in \(R^n\).

The ECE perspective helps anchor these abstract definitions. If \(A\) represents your system, then the column space tells you what outputs are physically possible. The null space tells you about inputs that don’t produce any detectable output, which can be critical for system design and troubleshooting.

The Column Space & System Consistency (\(A\mathbf{x} = \mathbf{b}\))

The column space directly tells us about the solvability of linear systems.

Recall that the matrix-vector product \(A\mathbf{x}\) can be expressed as a linear combination of the column vectors of \(A\):

\[ A\mathbf{x} = x_{1}\mathbf{c}_{1} + x_{2}\mathbf{c}_{2} + \dots + x_{n}\mathbf{c}_{n} \tag{1} \]

Thus, a linear system \(A\mathbf{x} = \mathbf{b}\) can be written as:

\[ x_{1}\mathbf{c}_{1} + x_{2}\mathbf{c}_{2} + \dots + x_{n}\mathbf{c}_{n} = \mathbf{b} \tag{2} \]

Important

THEOREM 4.7.1: A system of linear equations \(A\mathbf{x} = \mathbf{b}\) is consistent if and only if \(\mathbf{b}\) is in the column space of \(A\).

Tip

ECE Application: Reachability

• For a control system, if \(A\) represents the system dynamics and \(\mathbf{b}\) is a desired state, then \(\mathbf{b}\) must be in the column space of \(A\) to be reachable.
• If \(\mathbf{b}\) is not in \(\text{col}(A)\), it’s impossible to achieve that output state with the given system inputs.

This slide highlights one of the most direct and practical applications of the column space: determining if a linear system \(A\mathbf{x}=\mathbf{b}\) has a solution.

The key insight is that \(A\mathbf{x}\) is always a linear combination of the columns of \(A\). So, for a solution \(\mathbf{x}\) to exist, the vector \(\mathbf{b}\) must be expressible as a linear combination of \(A\)’s columns. This is precisely the definition of being in the column space.

In ECE, this translates to the concept of reachability. If you have a system, and you want it to produce a specific output or reach a certain state, that target output or state must be within the system’s column space. If it’s not, no matter what input you apply, you’ll never get there. This is fundamental in control system design.

Interactive Example: Forming b from Column Space

Let’s explore if \(\mathbf{b}\) can be formed by column vectors of \(A\). Adjust the coefficients \(x_1, x_2, x_3\) to see if you can match the target vector \(\mathbf{b}\).

\[ A = {\left[\begin{array}{l l l}{-1}&{3}&{2}\\ {1}&{2}&{-3}\\ {2}&{1}&{-2}\end{array}\right]}, \quad \mathbf{b_{target}} = {\left[\begin{array}{l}{1}\\ {-9}\\ {-3}\end{array}\right]} \]

\[ \mathbf{x} = \left[ \begin{array}{l}x_{1} \\ x_{2} \\ x_{3} \end{array} \right] \implies A\mathbf{x} = x_{1}\mathbf{c}_{1} + x_{2}\mathbf{c}_{2} + x_{3}\mathbf{c}_{3} = \mathbf{b_{current}} \]

Interactive Example: Forming b from Column Space

viewof x1 = Inputs.range([-5, 5], {value: 0, step: 0.1, label: "x1"});
viewof x2 = Inputs.range([-5, 5], {value: 0, step: 0.1, label: "x2"});
viewof x3 = Inputs.range([-5, 5], {value: 0, step: 0.1, label: "x3"});

Here’s an interactive demonstration of Theorem 4.7.1. We have a matrix \(A\) and a target vector \(\mathbf{b}\). Your task is to adjust the values of \(x_1, x_2,\) and \(x_3\) using the sliders. As you do, the blue vector, which represents \(A\mathbf{x}\) (the linear combination of \(A\)’s columns), will change.

The goal is to see if you can make the blue vector perfectly overlap with the red target vector \(\mathbf{b}\). If you can, it means \(\mathbf{b}\) is indeed in the column space of \(A\), and the system \(A\mathbf{x}=\mathbf{b}\) is consistent.

(Hint: The solution for \(x_1, x_2, x_3\) for this specific example is \(2, -1, 3\) respectively, as per Example 2 in the text. Try to find it!) This helps build intuition for how linear combinations work and what it means for a vector to “live” within the space spanned by other vectors.

The Null Space and General Solutions (\(A\mathbf{x} = \mathbf{0}\))

The null space is crucial for understanding the structure of all solutions to \(A\mathbf{x} = \mathbf{b}\).

Note

THEOREM 4.7.2: If \(\mathbf{x}_{0}\) is any particular solution of a consistent linear system \(A\mathbf{x} = \mathbf{b}\), and if \(S = \{\mathbf{v}_{1}, \ldots , \mathbf{v}_{k}\}\) is a basis for the null space of \(A\), then every solution of \(A\mathbf{x} = \mathbf{b}\) can be expressed in the form:

\[ \mathbf{x} = \mathbf{x}_{0} + c_{1}\mathbf{v}_{1} + c_{2}\mathbf{v}_{2} + \dots + c_{k}\mathbf{v}_{k} \tag{3} \]

Conversely, for all choices of scalars \(c_{1}, \ldots , c_{k}\), the vector \(\mathbf{x}\) in this formula is a solution of \(A\mathbf{x} = \mathbf{b}\).

This means the general solution of \(A\mathbf{x} = \mathbf{b}\) is the sum of:

1. A particular solution (\(\mathbf{x}_0\))
2. The general solution of the corresponding homogeneous system (\(A\mathbf{x} = \mathbf{0}\)), which is precisely the null space of \(A\).

Geometrically, the solution set of \(A\mathbf{x} = \mathbf{b}\) is a translation of the null space of \(A\) by the vector \(\mathbf{x}_0\).

While the column space tells us if a solution exists, the null space helps us understand the structure of all possible solutions.

Theorem 4.7.2 is incredibly powerful. It states that if you find just one solution to \(A\mathbf{x}=\mathbf{b}\) (that’s \(\mathbf{x}_0\)), then you can find all other solutions by adding vectors from the null space of \(A\).

Think of it this way: the null space represents all the “degrees of freedom” or “independent choices” you can make in your input \(\mathbf{x}\) that don’t affect the output \(\mathbf{b}\) (if you’re already at \(\mathbf{b}\)).

Geometrically, this is a beautiful concept. The null space is always a subspace passing through the origin. The set of all solutions to \(A\mathbf{x}=\mathbf{b}\) is just that same null space, but shifted, or translated, by a particular solution \(\mathbf{x}_0\). This is often visualized as a line or a plane that doesn’t necessarily pass through the origin.

Visualizing Solution Space: Null Space Translation

The solution set of \(A\mathbf{x} = \mathbf{b}\) is a translated version of the null space of \(A\).

Figure 4.7.1: The solution set of \(A\mathbf{x} = \mathbf{b}\) as a translation of the null space of \(A\).

Visualizing Solution Space: Null Space Translation

Interactive Example: Null Space Translation in 2D

Let \(A = [1 \quad -1]\). The homogeneous system \(A\mathbf{x} = \mathbf{0}\) is \(x_1 - x_2 = 0\), so \(x_1 = x_2\). The null space is the line \(y=x\) (i.e., \(\text{span}\{[1, 1]\}\)).

Consider \(A\mathbf{x} = \mathbf{b}\) with \(\mathbf{b} = [2]\). A particular solution is \(\mathbf{x}_0 = [2, 0]\) (since \(1(2) - 1(0) = 2\)). The general solution is \(\mathbf{x} = [2, 0] + c[1, 1]\).

viewof c_val = Inputs.range([-5, 5], {value: 0, step: 0.1, label: "Null Space Scalar (c)"});

This diagram visually represents the concept we just discussed. The null space of \(A\) is a subspace that passes through the origin. When you add a particular solution \(\mathbf{x}_0\) to every vector in the null space, you effectively shift, or translate, that entire subspace. The result is the complete set of solutions to \(A\mathbf{x}=\mathbf{b}\).

In the interactive example, we have a very simple 2D case. Our matrix \(A = [1 \quad -1]\) has a null space that is the line \(x_1 - x_2 = 0\), or \(x_1 = x_2\). This is the dashed gray line passing through the origin.

We picked a particular solution \(\mathbf{x}_0 = [2, 0]\) for the system \(A\mathbf{x} = [2]\). This is shown as the orange point.

The blue line represents the entire solution set for \(A\mathbf{x} = [2]\). You can see it’s parallel to the null space, but shifted by \(\mathbf{x}_0\). As you move the slider for c_val, the green star moves along this blue line, demonstrating that any point on this line is a valid solution and can be reached by adding a scalar multiple of the null space basis vector to \(\mathbf{x}_0\).

Impact of Elementary Row Operations

Elementary Row Operations (EROs) are powerful, but they have different effects on our key spaces.

Important

THEOREM 4.7.3 & 4.7.4:

• Elementary row operations do not change the null space of a matrix.
• Elementary row operations do not change the row space of a matrix.

Why EROs don’t change Null Space:

• EROs on \(A\) are equivalent to multiplying \(A\) by an invertible matrix \(E\) (i.e., \(EA\)).
• If \(A\mathbf{x} = \mathbf{0}\), then \(EA\mathbf{x} = E\mathbf{0} = \mathbf{0}\).
• So, the solutions to \(A\mathbf{x} = \mathbf{0}\) are the same as for \(EA\mathbf{x} = \mathbf{0}\).

Impact of Elementary Row Operations

Why EROs don’t change Row Space:

• EROs are linear combinations of rows.
• The new rows are still in the span of the original rows.
• Since EROs are reversible, the original rows are also in the span of the new rows.
• Thus, the span remains the same.

Warning

EROs DO change the Column Space of a matrix!

Consider: \[ A = {\left[ \begin{array}{l l}{1} & {3}\\ {2} & {6} \end{array} \right]} \quad \xrightarrow{R_2 \to R_2 - 2R_1} \quad B = {\left[ \begin{array}{l l}{1} & {3}\\ {0} & {0} \end{array} \right]} \]

• Column Space of \(A\): All multiples of \(\left[ \begin{array}{l}1 \\ 2 \end{array} \right]\) (e.g., \([1,2]^T, [2,4]^T\), etc.)
• Column Space of \(B\): All multiples of \(\left[ \begin{array}{l}1 \\ 0 \end{array} \right]\) (e.g., \([1,0]^T, [2,0]^T\), etc.)

These are different spaces!

This is a critical point that often trips up students. While EROs are fundamental for solving systems and simplifying matrices, they don’t affect all vector spaces equally.

The null space is preserved because EROs fundamentally don’t change the solution set of a homogeneous system. If \(\mathbf{x}\) makes \(A\mathbf{x}=\mathbf{0}\), it will also make the row-reduced version of \(A\) times \(\mathbf{x}\) equal to \(\mathbf{0}\).

Similarly for the row space: EROs are just linear combinations of rows. If you take a linear combination of the original rows, you’re still within the span. And since EROs are reversible, you can get the original rows back from the new ones, meaning the span is identical.

However, the column space does change! The example clearly shows this. The column space of \(A\) is a line through the origin with slope 2. After one ERO, the column space of \(B\) is the x-axis. These are distinct spaces. This means we cannot directly find a basis for the column space of \(A\) by just taking the column vectors of its row echelon form. We’ll see how to handle this soon.

Finding a Basis for the Null Space

The null space of \(A\) is the solution space of \(A\mathbf{x} = \mathbf{0}\).

EXAMPLE 4: Finding a Basis for the Null Space of a Matrix

Find a basis for the null space of the matrix:

\[ A = {\left[ \begin{array}{l l l l l l}{1} & {3} & {-2} & 0 & 2 & 0\\ {2} & {6} & {-5} & {-2} & 4 & {-3}\\ {0} & 0 & 5 & {10} & 0 & {15}\\ {2} & {6} & 0 & 8 & 4 & {18} \end{array} \right]} \]

Solution: We need to solve the homogeneous system \(A\mathbf{x} = \mathbf{0}\). (This involves reducing \(A\) to its row echelon form or reduced row echelon form, and then expressing the general solution in parametric vector form.)

Finding a Basis for the Null Space

After row reduction (as shown in Section 3.4), the general solution to \(A\mathbf{x} = \mathbf{0}\) is: \[ \mathbf{x} = \left[ \begin{array}{c}-3x_2 - 4x_4 - 2x_6 \\ x_2 \\ -2x_4 - 3x_6 \\ x_4 \\ x_5 \\ x_6 \end{array} \right] = x_2\left[ \begin{array}{c}-3 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \end{array} \right] + x_4\left[ \begin{array}{c}-4 \\ 0 \\ -2 \\ 1 \\ 0 \\ 0 \end{array} \right] + x_6\left[ \begin{array}{c}-2 \\ 0 \\ -3 \\ 0 \\ 0 \\ 1 \end{array} \right] \]

Thus, a basis for the null space of \(A\) is:

\[ \mathbf{v}_{1} = \left[ \begin{array}{c} - 3 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \end{array} \right], \quad \mathbf{v}_{2} = \left[ \begin{array}{c} - 4 \\ 0 \\ -2 \\ 1 \\ 0 \\ 0 \end{array} \right], \quad \mathbf{v}_{3} = \left[ \begin{array}{c} - 2 \\ 0 \\ -3 \\ 0 \\ 0 \\ 1 \end{array} \right] \]

To find a basis for the null space, we always start by solving the homogeneous system \(A\mathbf{x} = \mathbf{0}\). This involves reducing the augmented matrix \([A | \mathbf{0}]\) to its reduced row echelon form. Since the right-hand side is all zeros, we essentially just reduce \(A\).

Once in RREF, we identify the free variables. Each free variable corresponds to a basis vector for the null space. We express the leading variables in terms of the free variables, and then write the general solution in parametric vector form.

In this example, \(x_2, x_4, x_6\) were the free variables. By setting one free variable to 1 and the others to 0, we can systematically extract the basis vectors. This process gives us the three basis vectors shown, which span the null space of \(A\). The dimension of the null space, which is 3 in this case, is called the nullity of the matrix.

Bases for Row and Column Spaces (from RREF)

When a matrix is in row echelon form, finding bases becomes simpler.

Important

THEOREM 4.7.5:

If a matrix \(R\) is in row echelon form:

1. The row vectors with the leading 1’s (the nonzero row vectors) form a basis for the row space of \(R\).
2. The column vectors with the leading 1’s of the row vectors form a basis for the column space of \(R\).

Bases for Row and Column Spaces (from RREF)

EXAMPLE 5: Bases for Row and Column Spaces of a Matrix in Row Echelon Form

Find bases for the row and column spaces of the matrix:

\[ R = {\left[ \begin{array}{l l l l l}{1} & {-2} & {5} & {0} & {3}\\ {0} & {1} & {3} & {0} & {0}\\ {0} & {0} & {0} & {1} & {0}\\ {0} & {0} & {0} & {0} & {0} \end{array} \right]} \]

Solution:

• Basis for Row Space of \(R\): The nonzero row vectors are:

  \(\mathbf{r}_{1} = [1\quad -2\quad 5\quad 0\quad 3]\)

  \(\mathbf{r}_{2} = [0\quad 1\quad 3\quad 0\quad 0]\)

  \(\mathbf{r}_{3} = [0\quad 0\quad 0\quad 1\quad 0]\)

  These form a basis for \(\text{row}(R)\).

Bases for Row and Column Spaces (from RREF)

• Basis for Column Space of \(R\): The leading 1’s are in columns 1, 2, and 4. The corresponding column vectors of \(R\) are:

  \(\mathbf{c}_{1} = \left[ \begin{array}{c}1 \\ 0 \\ 0 \\ 0 \end{array} \right], \quad \mathbf{c}_{2} = \left[ \begin{array}{c} - 2 \\ 1 \\ 0 \\ 0 \end{array} \right], \quad \mathbf{c}_{4} = \left[ \begin{array}{c}0 \\ 0 \\ 1 \\ 0 \end{array} \right]\)

  These form a basis for \(\text{col}(R)\).

This theorem provides a very convenient way to find bases if your matrix is already in row echelon form.

For the row space, it’s straightforward: just take all the non-zero rows. Because of the leading 1’s and the zeros below them, these rows are guaranteed to be linearly independent, and they span the row space.

For the column space of \(R\), we look for the columns that contain the leading 1’s, often called pivot columns. The column vectors corresponding to these pivot positions form a basis for the column space of that row echelon matrix R.

It’s crucial to remember the distinction here: this theorem gives us bases for \(\text{row}(R)\) and \(\text{col}(R)\), not necessarily \(\text{col}(A)\) if \(R\) is just a row echelon form of \(A\). We’ll address how to get \(\text{col}(A)\) next.

Finding a Basis for the Row Space (of original \(A\))

Since EROs do not change the row space, we can find a basis for \(\text{row}(A)\) from its row echelon form.

EXAMPLE 6: Basis for a Row Space by Row Reduction

Find a basis for the row space of the matrix:

\[ A = {\left[ \begin{array}{l l l l l l}{1} & {-3} & {4} & {-2} & {5} & {4}\\ {2} & {-6} & {9} & {-1} & {8} & {2}\\ {2} & {-6} & {9} & {-1} & {9} & {7}\\ {-1} & {3} & {-4} & {2} & {-5} & {-4} \end{array} \right]} \]

Solution: Reduce \(A\) to row echelon form \(R\):

\[ R = {\left[ \begin{array}{l l l l l l}{1} & {-3} & {4} & {-2} & {5} & {4}\\ {0} & {0} & {1} & {3} & {-2} & {-6}\\ {0} & {0} & {0} & {0} & {1} & {5}\\ {0} & {0} & {0} & {0} & {0} & {0} \end{array} \right]} \]

Finding a Basis for the Row Space (of original \(A\))

By Theorem 4.7.5, the nonzero row vectors of \(R\) form a basis for the row space of \(R\). Since \(\text{row}(A) = \text{row}(R)\), these vectors also form a basis for \(\text{row}(A)\):

\[ \begin{array}{r l}{\mathbf{r}_{1}=[1}&{{}-3\qquad4\quad-2\qquad5\quad4]}\\ {\mathbf{r}_{2}=[0\qquad0\qquad1\quad3\quad-2\quad-6]}\\ {\mathbf{r}_{3}=[0\qquad0\qquad0\qquad0\quad1\quad5]}\end{array} \]

Because elementary row operations preserve the row space, finding a basis for the row space of any matrix \(A\) is straightforward:

1. Reduce \(A\) to any row echelon form \(R\).
2. The nonzero rows of \(R\) are your basis vectors for the row space of \(A\).

This method is reliable and easy to apply. Notice that these basis vectors are generally not the original row vectors of \(A\). They are linear combinations of them, but they still span the exact same space.

Finding a Basis for the Column Space (of original \(A\))

This is trickier, as EROs change the column space itself. However, they preserve linear dependence relations between columns.

Important

THEOREM 4.7.6: If \(A\) and \(B\) are row equivalent matrices, then:

1. A given set of column vectors of \(A\) is linearly independent if and only if the corresponding column vectors of \(B\) are linearly independent.
2. A given set of column vectors of \(A\) forms a basis for the column space of \(A\) if and only if the corresponding column vectors of \(B\) form a basis for the column space of \(B\).

This theorem is key! It tells us that while the column vectors themselves change, their relationships (linear independence/dependence) do not.

Finding a Basis for the Column Space (of original \(A\))

Method to find a basis for \(\text{col}(A)\) from original columns:

1. Reduce \(A\) to a row echelon form \(R\).
2. Identify the columns in \(R\) that contain the leading 1’s (pivot columns).
3. The corresponding columns in the original matrix \(A\) form a basis for \(\text{col}(A)\).

EXAMPLE 7: Basis for a Column Space by Row Reduction

Using matrix \(A\) from Example 6 and its RREF \(R\):

\[ A = {\left[ \begin{array}{r r r r r r}{\mathbf{c_1}} & {\mathbf{c_2}} & {\mathbf{c_3}} & {\mathbf{c_4}} & {\mathbf{c_5}} & {\mathbf{c_6}}\\ {1} & {-3} & {4} & {-2} & {5} & {4}\\ {2} & {-6} & {9} & {-1} & {8} & {2}\\ {2} & {-6} & {9} & {-1} & {9} & {7}\\ {-1} & {3} & {-4} & {2} & {-5} & {-4} \end{array} \right]} \quad \xrightarrow{\text{EROs}} \quad R={\left[\begin{array}{r r r r r r}{1}&{-3}&{4}&{-2}&{5}&{4}\\ {0}&{0}&{1}&{3}&{-2}&{-6}\\ {0}&{0}&{0}&{0}&{1}&{5}\\ {0}&{0}&{0}&{0}&{0}&{0}\end{array}\right]} \]

Finding a Basis for the Column Space (of original \(A\))

1. Pivot columns in \(R\): Columns 1, 3, and 5 contain leading 1’s.
2. Corresponding columns in \(A\): \(\mathbf{c}_{1}, \mathbf{c}_{3}, \mathbf{c}_{5}\).

Thus, the basis for \(\text{col}(A)\) consists of:

\[ \mathbf{c}_{1}={\left[\begin{array}{l}{1}\\ {2}\\ {2}\\ {-1}\end{array}\right]},\quad\mathbf{c}_{3}={\left[\begin{array}{l}{4}\\ {9}\\ {9}\\ {-4}\end{array}\right]},\quad\mathbf{c}_{5}={\left[\begin{array}{l}{5}\\ {8}\\ {9}\\ {-5}\end{array}\right]} \]

This is where the subtlety comes in. We established that EROs change the column space. So, we can’t just take the columns of \(R\) that have leading 1’s as a basis for \(\text{col}(A)\).

However, Theorem 4.7.6 is our savior. It tells us that if a set of columns is linearly independent in \(A\), their corresponding columns in \(R\) are also linearly independent, and vice versa. This means the pattern of linear dependence and independence is preserved.

So, the strategy is:

1. Reduce \(A\) to \(R\).
2. Find the pivot columns in \(R\). These are the columns that would form a basis for \(\text{col}(R)\).
3. Then, go back to the original matrix \(A\) and pick out the columns that correspond to those pivot positions. These original columns will form a basis for \(\text{col}(A)\).

Example 7 clearly demonstrates this. Columns 1, 3, and 5 are pivot columns in \(R\). Therefore, columns 1, 3, and 5 from the original matrix A form a basis for \(\text{col}(A)\).

Interactive Column Space Basis Finder

Provide a matrix \(A\) and let Python find its RREF and identify a basis for its column space.

Enter your matrix \(A\) (e.g., [[1, 2, 3], [4, 5, 6], [7, 8, 9]]):

viewof matrix_input_cs = Inputs.textarea({value: "[[1, -3, 4, -2, 5, 4], [2, -6, 9, -1, 8, 2], [2, -6, 9, -1, 9, 7], [-1, 3, -4, 2, -5, -4]]", rows: 4, label: "Matrix A (as nested list)"});

This interactive tool allows you to input any matrix and immediately see the steps to find a basis for its column space.

1. Input your matrix \(A\) as a nested list (e.g., [[1, 2], [3, 4]]).
2. The Python code, using the SymPy library, calculates the Reduced Row Echelon Form (RREF) of your matrix.
3. It then identifies the pivot columns in the RREF.
4. Finally, it extracts the corresponding columns from your original matrix A to form the basis for the column space of A.

This is a direct application of the method we just discussed and can be very helpful for quickly checking your work or experimenting with different matrices.

Basis for Space Spanned by a Set of Vectors

The problem of finding a basis for the subspace spanned by a set of vectors is equivalent to finding a basis for the column space of a matrix.

Problem: Given a set of vectors \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{k}\}\) in \(R^{n}\), find a subset of these vectors that forms a basis for \(\operatorname {span}(S)\).

Solution:

1. Form a matrix \(A\) whose columns are the vectors \(\mathbf{v}_{1}, \ldots, \mathbf{v}_{k}\).
2. The column space of \(A\), \(\text{col}(A)\), is precisely \(\operatorname {span}(S)\).
3. Use the method from Example 7 (and Theorem 4.7.6) to find a basis for \(\text{col}(A)\) consisting of original column vectors.

Basis for Space Spanned by a Set of Vectors

EXAMPLE 8: Basis for the Space Spanned by a Set of Vectors

The following vectors span a subspace of \(R^{4}\). Find a subset of these vectors that forms a basis of this subspace.

\[ \begin{array}{r l} & {\mathbf{v}_{1} = (1,2,2, - 1),\qquad \mathbf{v}_{2} = (-3, - 6, - 6,3),}\\ & {\mathbf{v}_{3} = (4,9,9, - 4),\qquad \mathbf{v}_{4} = (-2, - 1, - 1,2),}\\ & {\mathbf{v}_{5} = (5,8,9, - 5),\qquad \mathbf{v}_{6} = (4,2,7, - 4)} \end{array} \]

Solution: If we write these vectors as columns, we get the matrix \(A\) from Example 7:

\[ A = [\mathbf{v}_{1} \quad \mathbf{v}_{2} \quad \mathbf{v}_{3} \quad \mathbf{v}_{4} \quad \mathbf{v}_{5} \quad \mathbf{v}_{6}] = {\left[ \begin{array}{r r r r r r}{1} & {-3} & {4} & {-2} & {5} & {4}\\ {2} & {-6} & {9} & {-1} & {8} & {2}\\ {2} & {-6} & {9} & {-1} & {9} & {7}\\ {-1} & {3} & {-4} & {2} & {-5} & {-4} \end{array} \right]} \]

From Example 7, the pivot columns in the RREF of \(A\) corresponded to columns 1, 3, and 5. Therefore, the vectors \(\mathbf{v}_{1}, \mathbf{v}_{3}\), and \(\mathbf{v}_{5}\) form a basis for \(\operatorname {span}\{\mathbf{v}_{1},\ldots,\mathbf{v}_{6}\}\).

This is a common type of problem: you’re given a set of vectors, and you need to find a smaller, linearly independent subset of them that still spans the same space. This is essentially finding a basis for the span of those vectors.

The trick is to realize that “the span of a set of vectors” is exactly the same as “the column space of the matrix formed by those vectors.” So, we can simply apply the method for finding a basis for the column space of a matrix.

As shown in Example 8, by constructing matrix \(A\) with the given vectors as columns, and then using the results from Example 7, we can quickly identify which of the original vectors form a basis for their span. This is a very efficient way to solve this kind of problem.

Basis for the Row Space (from original rows)

Sometimes, we need a basis for the row space that consists entirely of row vectors from the original matrix \(A\).

Method:

1. Transpose \(A\) to get \(A^T\). The row space of \(A\) is the column space of \(A^T\).
2. Find a basis for the column space of \(A^T\) using the method from Example 7 (i.e., reduce \(A^T\) to RREF, identify pivot columns, and take the corresponding columns from \(A^T\)).
3. Transpose these basis column vectors back to row vectors. These will be original row vectors of \(A\) that form a basis for \(\text{row}(A)\).

Basis for the Row Space (from original rows)

EXAMPLE 9: Basis for the Row Space of a Matrix (using original rows)

Find a basis for the row space of \(A\) consisting entirely of row vectors from \(A\):

\[ A = {\left[ \begin{array}{l l l l l}{1} & {-2} & 0 & 0 & 3\\ 2 & {-5} & {-3} & {-2} & 6\\ 0 & 5 & {15} & {10} & 0\\ 2 & 6 & {18} & 8 & 6 \end{array} \right]} \]

Solution:

1. Transpose \(A\): \[ A^{T} = {\left[ \begin{array}{l l l l}{1} & {2} & 0 & 2\\ {-2} & {-5} & 5 & 6\\ 0 & {-3} & {15} & {18}\\ 0 & {-2} & {10} & 8\\ 3 & 6 & 0 & 6 \end{array} \right]} \]

Basis for the Row Space (from original rows)

2. Reduce \(A^T\) to RREF: (After row operations) \[ R_{A^T} = {\left[ \begin{array}{l l l l}{1} & {2} & 0 & 2\\ 0 & 1 & {-5} & {-10}\\ 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 \end{array} \right]} \]
3. Identify pivot columns in \(R_{A^T}\): Columns 1, 2, and 4.
4. Corresponding columns from \(A^T\): These are \(\mathbf{c}_{1}, \mathbf{c}_{2}, \mathbf{c}_{4}\) of \(A^T\). \[ \mathbf{c}_{1} = {\left[ \begin{array}{l}{1}\\ {-2}\\ {0}\\ {0}\\ {3} \end{array} \right]},\quad \mathbf{c}_{2} = {\left[ \begin{array}{l}{2}\\ {-5}\\ {-3}\\ {-2}\\ {6} \end{array} \right]},\quad \mathbf{c}_{4} = {\left[ \begin{array}{l}{2}\\ {6}\\ {18}\\ {8}\\ {6} \end{array} \right]} \]

Basis for the Row Space (from original rows)

5. Transpose back to row vectors: These are the original rows \(\mathbf{r}_1, \mathbf{r}_2, \mathbf{r}_4\) of \(A\). \[ \mathbf{r}_{1} = [1 \quad -2\quad 0\quad 0\quad 3],\quad \mathbf{r}_{2} = [2 \quad -5 \quad -3 \quad -2\quad 6],\quad \mathbf{r}_{4} = [2\quad 6\quad 18\quad 8\quad 6] \] These form a basis for \(\text{row}(A)\).

We’ve seen how to get a basis for the row space from the nonzero rows of the RREF. But sometimes, for interpretability or other reasons, we might need a basis that consists only of actual rows from the original matrix \(A\).

This method uses a clever trick: the row space of \(A\) is the same as the column space of its transpose, \(A^T\). So, we transpose \(A\), apply the method for finding a basis for the column space (which gives us original columns of \(A^T\)), and then transpose those back to get original rows of \(A\).

Example 9 walks through this process. We transpose \(A\), reduce \(A^T\) to RREF, identify pivot columns, pick those columns from the original \(A^T\), and then transpose them back to get the basis vectors for \(\text{row}(A)\) which are indeed rows from the original matrix \(A\). This is a powerful technique for specific applications where the original row vectors are preferred for the basis.

Basis and Linear Combinations (Part a)

This problem combines finding a basis with expressing other vectors as linear combinations.

Problem: Given a set of vectors \(S = \{\mathbf{v}_{1}, \ldots , \mathbf{v}_{k}\}\), find a subset that forms a basis for \(\operatorname {span}(S)\), and express each vector not in that basis as a linear combination of the basis vectors.

Basis and Linear Combinations (Part a)

EXAMPLE 10: Basis and Linear Combinations

1. Find a subset of the vectors below that forms a basis for their span:

\[ \begin{array}{c}{{\mathbf{v}_{1}=(1,-2,0,3),\quad\mathbf{v}_{2}=(2,-5,-3,6),}}\\ {{\mathbf{v}_{3}=(0,1,3,0),\quad\mathbf{v}_{4}=(2,-1,4,-7),\quad\mathbf{v}_{5}=(5,-8,1,2)}}\end{array} \]

Solution (a): 1. Form matrix \(A\) with vectors as columns: \[ A = \begin{array}{r}{\left[ \begin{array}{c c c c c}{1} & {2} & {0} & {2} & {5}\\ {-2} & {-5} & {1} & {-1} & {-8}\\ {0} & {-3} & {3} & {4} & {1}\\ {3} & {6} & {0} & {-7} & {2} \end{array} \right]}\\ {\uparrow \quad \uparrow \quad \uparrow \quad \uparrow \quad \uparrow}\\ {\mathbf{v}_{1}\quad \mathbf{v}_{2}\quad \mathbf{v}_{3}\quad \mathbf{v}_{4}\quad \mathbf{v}_{5}} \end{array} \]

Basis and Linear Combinations (Part a)

2. Reduce \(A\) to Reduced Row Echelon Form (RREF) \(R\): \[ R = \begin{array}{r}{\left[ \begin{array}{c c c c c}{1} & {0} & {2} & {0} & {1}\\ {0} & {1} & {-1} & {0} & {1}\\ {0} & {0} & {0} & {1} & {1}\\ {0} & {0} & {0} & {0} & {0} \end{array} \right]}\\ {\uparrow \quad \uparrow \quad \uparrow \quad \uparrow \quad \uparrow \quad \uparrow}\\ {\mathbf{w}_{1}\quad \mathbf{w}_{2}\quad \mathbf{w}_{3}\quad \mathbf{w}_{4}\quad \mathbf{w}_{5}} \end{array} \]
3. Identify pivot columns in \(R\): Columns 1, 2, and 4.
4. Corresponding columns from original \(A\): \(\mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{4}\). Thus, \(\{\mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{4}\}\) is a basis for the subspace spanned by the given vectors.

This example tackles a more comprehensive problem. Not only do we need to find a basis, but we also need to show how the “extra” vectors relate to that basis.

Part (a) is a direct application of what we just learned:

1. We construct a matrix where the given vectors are its columns.
2. We then reduce this matrix to its Reduced Row Echelon Form (RREF). The RREF is particularly useful for part (b).
3. We identify the pivot columns in the RREF. These tell us which columns in the original matrix form a basis. In this case, columns 1, 2, and 4 are pivot columns in \(R\), so \(\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_4\) form a basis for the span of all five vectors.

Basis and Linear Combinations (Part b)

Express each vector not in the basis as a linear combination of the basis vectors.

Solution (b): From the RREF \(R\) (columns \(\mathbf{w}_{1}, \ldots, \mathbf{w}_{5}\)):

\[ R = \begin{array}{r}{\left[ \begin{array}{c c c c c}{1} & {0} & {2} & {0} & {1}\\ {0} & {1} & {-1} & {0} & {1}\\ {0} & {0} & {0} & {1} & {1}\\ {0} & {0} & {0} & {0} & {0} \end{array} \right]}\\ {\uparrow \quad \uparrow \quad \uparrow \quad \uparrow \quad \uparrow \quad \uparrow}\\ {\mathbf{w}_{1}\quad \mathbf{w}_{2}\quad \mathbf{w}_{3}\quad \mathbf{w}_{4}\quad \mathbf{w}_{5}} \end{array} \]

The non-pivot columns are \(\mathbf{w}_3\) and \(\mathbf{w}_5\). We express them as linear combinations of the pivot columns \(\mathbf{w}_1, \mathbf{w}_2, \mathbf{w}_4\).

• For \(\mathbf{w}_3\): The third column of \(R\) is \([2, -1, 0, 0]^T\). This means \(\mathbf{w}_3 = 2\mathbf{w}_1 - 1\mathbf{w}_2\). (Since \(\mathbf{w}_1 = [1,0,0,0]^T\) and \(\mathbf{w}_2 = [0,1,0,0]^T\)).

• For \(\mathbf{w}_5\): The fifth column of \(R\) is \([1, 1, 1, 0]^T\). This means \(\mathbf{w}_5 = 1\mathbf{w}_1 + 1\mathbf{w}_2 + 1\mathbf{w}_4\). (Since \(\mathbf{w}_1 = [1,0,0,0]^T\), \(\mathbf{w}_2 = [0,1,0,0]^T\), and \(\mathbf{w}_4 = [0,0,1,0]^T\)).

These are the “dependency equations” in terms of the RREF columns.

Since the linear dependence relationships are preserved by EROs (Theorem 4.7.6), we can replace \(\mathbf{w}_i\) with \(\mathbf{v}_i\):

• \(\mathbf{v}_{3} = 2\mathbf{v}_{1} - \mathbf{v}_{2}\)
• \(\mathbf{v}_{5} = \mathbf{v}_{1} + \mathbf{v}_{2} + \mathbf{v}_{4}\)

Part (b) leverages the power of the Reduced Row Echelon Form. The RREF doesn’t just tell us which columns are pivots; it directly gives us the coefficients for expressing the non-pivot columns as linear combinations of the pivot columns.

For example, looking at \(\mathbf{w}_3\), its entries \([2, -1, 0, 0]^T\) directly tell us that \(\mathbf{w}_3 = 2\mathbf{w}_1 - 1\mathbf{w}_2\). This is because \(\mathbf{w}_1\) and \(\mathbf{w}_2\) are standard basis vectors within the pivot columns’ subspace. Similarly for \(\mathbf{w}_5\).

Crucially, because EROs preserve these linear dependence relationships, the same coefficients apply to the original vectors. So, if \(\mathbf{w}_3 = 2\mathbf{w}_1 - \mathbf{w}_2\), then \(\mathbf{v}_3 = 2\mathbf{v}_1 - \mathbf{v}_2\). This is a very elegant and efficient way to express non-basis vectors as linear combinations of the basis vectors.

Summary of Basis Finding Steps

Tip

General Procedure for Basis and Linear Combinations:

1. Form Matrix: Create matrix \(A\) with the given vectors \(\{\mathbf{v}_{1}, \ldots , \mathbf{v}_{k}\}\) as its columns.
2. Reduce to RREF: Transform \(A\) into its Reduced Row Echelon Form \(R\).
3. Identify \(R\) Columns: Denote the columns of \(R\) as \(\mathbf{w}_{1}, \ldots , \mathbf{w}_{k}\).
4. Basis for Span: The columns of \(A\) corresponding to the pivot columns (those with leading 1’s) in \(R\) form a basis for \(\operatorname {span}(\{\mathbf{v}_{i}\})\).
5. Dependency Equations (for \(R\)): For each non-pivot column \(\mathbf{w}_j\) in \(R\), express it as a linear combination of the pivot columns \(\mathbf{w}_p\) that precede it. The coefficients are directly visible in \(\mathbf{w}_j\).
6. Dependency Equations (for \(A\)): Translate these dependency equations by replacing each \(\mathbf{w}_i\) with its corresponding original vector \(\mathbf{v}_i\).

This slide provides a concise summary of the entire process we just covered for finding a basis from a set of vectors and expressing the non-basis vectors as linear combinations. This six-step procedure is a powerful algorithm you can apply to many problems.

Remember, the key is using the RREF. It simplifies the matrix enough to easily identify linear independence (pivot columns) and to read off the coefficients for linear combinations (from the entries of the non-pivot columns).

ECE Application: Control Systems and State Space

These vector spaces are fundamental in State-Space Control Systems.

System Model: \[ \dot{\mathbf{x}}(t) = A\mathbf{x}(t) + B\mathbf{u}(t) \\ \mathbf{y}(t) = C\mathbf{x}(t) + D\mathbf{u}(t) \]

• \(\mathbf{x}(t)\): State vector
• \(\mathbf{u}(t)\): Input vector
• \(\mathbf{y}(t)\): Output vector
• \(A, B, C, D\): System matrices

ECE Application: Control Systems and State Space

How the spaces apply:

• Column Space of \(A\) (or \(B\)):
  • Reachability/Controllability: Can the system reach any desired state \(\mathbf{x}\) from its inputs \(\mathbf{u}\)? This relates to the column space of the controllability matrix.
• Null Space of \(C\):
  • Observability: What states \(\mathbf{x}\) produce zero output \(\mathbf{y}\)? These are unobservable states.
• Row Space of \(A\) (or \(C\)):
  • Observability: What information about the state \(\mathbf{x}\) is contained in the output \(\mathbf{y}\)? This relates to the row space of the observability matrix.

For ECE students, one of the most powerful applications of these concepts is in control systems, particularly within the state-space framework.

Here, a system’s behavior is described by its state, input, and output vectors, and the matrices \(A, B, C, D\) define the system dynamics.

• The column space of B (or more generally, the controllability matrix) tells us if we can steer the system from any initial state to any desired final state using our inputs. If a state isn’t in this column space, it’s unreachable.
• The null space of C helps us understand observability. If a state \(\mathbf{x}\) is in the null space of \(C\), it means that state produces zero output. We cannot “see” or deduce information about such states from the system’s outputs. These are unobservable states.
• The row space of C (or the observability matrix) complements this, telling us what information about the state is contained in the output.

These concepts are critical for designing robust and effective control systems, ensuring that we can both control the system as intended and monitor its behavior effectively.

Conclusion & Key Takeaways

What we’ve learned:

• Row Space (\(\text{row}(A)\)): All linear combinations of \(A\)’s rows. Lives in \(R^n\).
  • Basis from nonzero rows of RREF.
• Column Space (\(\text{col}(A)\)): All linear combinations of \(A\)’s columns. Lives in \(R^m\).
  • Basis from original columns of \(A\) corresponding to pivot columns in RREF.
  • Determines system consistency (\(A\mathbf{x}=\mathbf{b}\) has a solution iff \(\mathbf{b} \in \text{col}(A)\)).
• Null Space (\(\text{null}(A)\)): All solutions to \(A\mathbf{x}=\mathbf{0}\). Lives in \(R^n\).
  • Basis from parametric vector form of general solution to \(A\mathbf{x}=\mathbf{0}\).
  • Translates to generate all solutions for \(A\mathbf{x}=\mathbf{b}\).

Conclusion & Key Takeaways

Key Relationships:

• EROs preserve row space and null space, but change column space.
• EROs preserve linear dependency relationships among columns.

Important

Why this matters for ECE: - System Capabilities: \(\text{col}(A)\) defines what outputs/states your system can achieve. - System Blind Spots: \(\text{null}(A)\) reveals inputs that are ‘ignored’ or states that are ‘unobservable’. - Efficiency: Bases help identify the minimal set of independent components needed to describe a system’s behavior.

Tip

Think of it as diagnosing a circuit: - Can I get this specific output voltage (is it in the column space)? - What inputs cause no current flow (are they in the null space)? - What are the fundamental independent current/voltage loops (basis for row/column space)?

To wrap up, we’ve covered three incredibly important vector spaces associated with matrices: the row space, column space, and null space.

Each space provides a unique lens through which to understand the properties and behavior of a matrix, and by extension, a linear system.

• The column space answers questions about what outputs are possible.
• The null space answers questions about what inputs are ‘neutralized’ or what states are unobservable.
• The row space provides insights into the input-output mapping.

For ECE, these aren’t just theoretical constructs. They are practical tools for analyzing, designing, and troubleshooting systems, whether you’re working with circuits, control systems, signal processing, or data analysis. Mastering these concepts will give you a deeper and more intuitive understanding of how linear systems truly operate.

Keep practicing with examples and consider how these spaces apply to the specific ECE problems you encounter. Thank you!