Linear Algebra

4.5 Dimension

Imron Rosyadi

Dimension

Quantifying the “Size” of a Vector Space

Welcome to this session on Dimension, a core concept in Linear Algebra. In the previous section, we established what a basis is. Now, we’ll connect the number of vectors in a basis to our intuitive understanding of dimension – like how a line is one-dimensional, and a plane is two-dimensional. This formal definition of dimension is crucial for understanding the structure and properties of vector spaces, which are fundamental in various ECE applications from signal processing to control theory.

The Concept of Dimension

The number of vectors in a basis is intrinsically linked to the “dimension” of a vector space.

Our goal is to make this intuitive notion precise.

THEOREM 4.5.1

All bases for a finite-dimensional vector space have the same number of vectors.

This theorem is foundational; it means the “number of vectors in a basis” is a well-defined property of the vector space itself, not just a property of a particular basis.

We start with a very important theorem: Theorem 4.5.1. It states that no matter which basis you choose for a given finite-dimensional vector space, they will all contain the same number of vectors. This is a powerful statement because it allows us to define the dimension of a vector space unambiguously. Before we formally define dimension, let’s look at a supporting theorem that helps prove this idea.

Supporting Theorem: Properties of Sets in \(n\)-Dimensional Space

To prove Theorem 4.5.1, we rely on properties of sets of vectors relative to a basis.

THEOREM 4.5.2

Let \(V\) be an \(n\)-dimensional vector space, and let \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{n}\}\) be any basis.

1. If a set in \(V\) has more than \(n\) vectors, then it is linearly dependent.
   
2. If a set in \(V\) has fewer than \(n\) vectors, then it does not span \(V\).

Proof of Theorem 4.5.1: If \(S_1\) and \(S_2\) are two bases for \(V\) with \(n_1\) and \(n_2\) vectors respectively. If \(n_1 > n_2\), then by Theorem 4.5.2(a), \(S_1\) would be linearly dependent, which contradicts it being a basis. If \(n_1 < n_2\), then by Theorem 4.5.2(b), \(S_1\) would not span \(V\), which also contradicts it being a basis. Therefore, \(n_1\) must equal \(n_2\).

Theorem 4.5.2 provides the logical stepping stones for proving Theorem 4.5.1. Part (a) says you can’t have “too many” linearly independent vectors in a space; if you exceed the dimension, your set must be dependent. Part (b) says you can’t have “too few” vectors to span the space; if you’re below the dimension, you won’t cover the whole space. Together, these two parts imply that any basis must have exactly \(n\) vectors, thus proving Theorem 4.5.1. The formal proof details can be found in your textbook.

Definition of Dimension

With Theorem 4.5.1 established, we can now formally define dimension.

DEFINITION 1

The dimension of a finite-dimensional vector space \(V\) is denoted by \(\dim (V)\) and is defined to be the number of vectors in a basis for \(V\). In addition, the zero vector space is defined to have dimension zero.

This definition aligns with our intuitive understanding of dimension.

This is the moment we’ve been building towards. The dimension of a finite-dimensional vector space is simply the number of vectors in any of its bases. Because all bases have the same number of vectors, this definition is robust. For completeness, the zero vector space, containing only the zero vector, is defined to have dimension zero, as its basis is the empty set.

EXAMPLE 1: Dimensions of Some Familiar Vector Spaces

Let’s apply the definition to some common vector spaces.

• \(\dim (R^{n}) = n\) The standard basis \(\{\mathbf{e}_{1}, \ldots, \mathbf{e}_{n}\}\) has \(n\) vectors. Example: \(\dim (R^3) = 3\).

• \(\dim (P_{n}) = n + 1\) The standard basis \(\{1, x, x^{2}, \ldots , x^{n}\}\) has \(n + 1\) vectors. Example: \(\dim (P_2) = 3\) (basis \(\{1, x, x^2\}\)).

• \(\dim (M_{mn}) = mn\) The standard basis (matrices with a single ‘1’ and rest ‘0’) has \(mn\) vectors. Example: \(\dim (M_{22}) = 4\) (basis \(M_1, M_2, M_3, M_4\)).

• \(\dim (\{\mathbf{0}\}) = 0\) The zero vector space has no non-zero vectors, so its basis is the empty set.

Here are some concrete examples of dimensions for spaces we’ve previously discussed. \(R^n\) has dimension \(n\), which makes perfect sense. The space of polynomials of degree \(n\) or less, \(P_n\), has dimension \(n+1\) because its standard basis includes the constant term and powers of x up to \(x^n\). Similarly, the space of \(m \times n\) matrices, \(M_{mn}\), has dimension \(mn\). These examples reinforce how the number of basis vectors directly translates to our concept of dimension.

EXAMPLE 2: Dimension of \(\operatorname{span}(S)\)

If a set of vectors \(S\) is linearly independent, it forms a basis for the space it spans.

If \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{r}\}\) is a linearly independent set, then: \[ \dim [\operatorname {span}\{\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots , \mathbf{v}_{r}\} ] = r \]

In words: The dimension of the space spanned by a linearly independent set of vectors is equal to the number of vectors in that set.

This is a very useful property. If you have a set of vectors that you know are linearly independent, then the dimension of the subspace they span is simply the number of vectors in that set. This simplifies finding the dimension of a subspace once you’ve identified a linearly independent spanning set for it.

EXAMPLE 3: Dimension of a Solution Space

Find a basis for and the dimension of the solution space of the homogeneous system:

\[ \begin{array}{r l r} & {} & {x_{1} + 3x_{2} - 2x_{3}\qquad +2x_{5}\qquad = 0}\\ & {} & {2x_{1} + 6x_{2} - 5x_{3} - 2x_{4} + 4x_{5} - 3x_{6} = 0}\\ & {} & {5x_{3} + 10x_{4}\qquad +15x_{6} = 0}\\ & {} & {2x_{1} + 6x_{2}\qquad +8x_{4} + 4x_{5} + 18x_{6} = 0} \end{array} \]

Let’s tackle a practical problem: finding the dimension of the solution space for a homogeneous system of linear equations. This is a common task in control systems and circuit analysis. We first need to find the general solution and express it in vector form.

EXAMPLE 3: Solution Space (Solution)

The solution to the system is given by: \(x_{1} = -3r - 4s - 2t\), \(x_{2} = r\), \(x_{3} = -2s\), \(x_{4} = s\), \(x_{5} = t\), \(x_{6} = 0\)

This can be written in vector form as: \[ (x_{1},x_{2},x_{3},x_{4},x_{5},x_{6}) = r(-3,1,0,0,0,0) + s(-4,0, - 2,1,0,0) + t(-2,0,0,0,1,0) \]

The vectors that span the solution space are: \[ \mathbf{v}_{1} = (-3,1,0,0,0,0),\quad \mathbf{v}_{2} = (-4,0, - 2,1,0,0),\quad \mathbf{v}_{3} = (-2,0,0,0,1,0) \]

These vectors are linearly independent.
Thus, the solution space has dimension 3.

By parameterizing the free variables (\(r, s, t\)), we can express any solution vector as a linear combination of these three specific vectors: \(\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\). These vectors inherently span the solution space. A quick inspection shows they are linearly independent because each vector has a ‘1’ in a position where the others have ‘0’ (e.g., \(x_2\) for \(\mathbf{v}_1\), \(x_4\) for \(\mathbf{v}_2\), \(x_5\) for \(\mathbf{v}_3\)). Since there are three linearly independent vectors that span the solution space, the dimension of the solution space is 3.

Interactive Example: Solution Space Basis

Let’s verify the linear independence of the spanning vectors using Python.
We’ll form a matrix with these vectors and check its rank.

Here’s a quick Python check. We form a matrix with our candidate basis vectors as rows. The rank of this matrix tells us the maximum number of linearly independent rows (or columns). Since the rank is 3, which is equal to the number of vectors, it confirms that these vectors are linearly independent. Thus, they form a basis for the solution space, and its dimension is 3. This method is generally applicable for checking linear independence.

Some Fundamental Theorems

We will now look at theorems that reveal interrelationships among linear independence, spanning sets, basis, and dimension.

Note

Informal Idea:
- Add a vector not in the span of a linearly independent set, and the set remains independent. - Remove a redundant vector from a spanning set, and it still spans the same space.

Figure 4.5.1: Illustrating adding/removing vectors.

These theorems are not just theoretical exercises; they are essential for understanding how to construct and manipulate bases, and how linear independence and spanning are connected. The “Plus/Minus Theorem” is particularly intuitive: if you have a set of vectors, you can either add a new, independent direction to make the set larger and still independent, or remove a redundant vector without shrinking the space it spans. This process is like fine-tuning a set of tools to be just right – not too many, not too few.

THEOREM 4.5.3: Plus/Minus Theorem

This theorem formally describes how adding or removing vectors affects linear independence and spanning.

Plus/Minus Theorem

Let \(S\) be a nonempty set of vectors in a vector space \(V\).

1. If \(S\) is a linearly independent set, and if \(\mathbf{v}\) is a vector in \(V\) that is outside of \(\operatorname{span}(S)\), then the set \(S\cup \{\mathbf{v}\}\) that results by inserting \(\mathbf{v}\) into \(S\) is still linearly independent.

2. If \(\mathbf{v}\) is a vector in \(S\) that is expressible as a linear combination of other vectors in \(S\), and if \(S - \{\mathbf{v}\}\) denotes the set obtained by removing \(\mathbf{v}\) from \(S\), then \(S\) and \(S - \{\mathbf{v}\}\) span the same space; that is, \(\operatorname{span}(S) = \operatorname{span}(S - \{\mathbf{v}\})\).

Part (a) is about growing a linearly independent set. If you pick a vector that cannot be formed by your current independent set, adding it will keep the expanded set linearly independent. Part (b) is about shrinking a spanning set without losing coverage. If one vector in your set is redundant (can be formed by the others), you can remove it, and the remaining vectors will still span the same space. These are powerful tools for adjusting sets of vectors to meet basis criteria.

EXAMPLE 4: Applying the Plus/Minus Theorem

Show that \(\mathbf{p}_{1} = 1 - x^{2}\), \(\mathbf{p}_{2} = 2 - x^{2}\), and \(\mathbf{p}_{3} = x^{3}\) are linearly independent vectors.

Solution:

1. Consider \(S_1 = \{\mathbf{p}_{1}, \mathbf{p}_{2}\}\). \(\mathbf{p}_{1} = 1 - x^2\) \(\mathbf{p}_{2} = 2 - x^2\) Neither is a scalar multiple of the other, so \(S_1\) is linearly independent.

2. Consider \(\mathbf{p}_{3} = x^{3}\). Can \(x^3\) be expressed as \(c_1(1-x^2) + c_2(2-x^2)\)? No, because the linear combination of \(\mathbf{p}_{1}\) and \(\mathbf{p}_{2}\) will only produce polynomials of degree at most 2, while \(\mathbf{p}_{3}\) is degree 3. Thus, \(\mathbf{p}_{3}\) is outside of \(\operatorname{span}(S_1)\).

Conclusion:

By Theorem 4.5.3(a), since \(S_1\) is linearly independent and \(\mathbf{p}_{3}\) is outside of \(\operatorname{span}(S_1)\), the set \(S_1 \cup \{\mathbf{p}_{3}\} = \{\mathbf{p}_{1}, \mathbf{p}_{2}, \mathbf{p}_{3}\}\) is also linearly independent.

This method allows us to build up linearly independent sets step-by-step.

Let’s apply the Plus/Minus Theorem to polynomials. We start with \(\mathbf{p}_1\) and \(\mathbf{p}_2\). They are clearly linearly independent as one is not a scalar multiple of the other. Now, we consider \(\mathbf{p}_3 = x^3\). The span of \(\mathbf{p}_1\) and \(\mathbf{p}_2\) only contains polynomials of degree 2 or less. Since \(\mathbf{p}_3\) is a polynomial of degree 3, it cannot be in the span of \(\{\mathbf{p}_1, \mathbf{p}_2\}\). Therefore, by part (a) of the Plus/Minus Theorem, adding \(\mathbf{p}_3\) to the set keeps the entire set linearly independent. This is a neat way to establish linear independence.

THEOREM 4.5.4: Basis by Inspection

If you know the dimension of a vector space, you only need to check one condition for a set of vectors to be a basis.

THEOREM 4.5.4

Let \(V\) be an \(n\)-dimensional vector space, and let \(S\) be a set in \(V\) with exactly \(n\) vectors. Then \(S\) is a basis for \(V\) if and only if \(S\) spans \(V\) or \(S\) is linearly independent.

Proof Logic: This theorem is powerful because it uses Theorem 4.5.2 and 4.5.3. If \(S\) spans \(V\) but isn’t a basis, it must be linearly dependent. Then, by 4.5.3(b), you could remove a vector, leaving \(n-1\) vectors that still span \(V\), which contradicts 4.5.2(b). A similar argument holds if \(S\) is linearly independent but doesn’t span \(V\).

Theorem 4.5.4 is a huge time-saver. If you already know the dimension of a vector space, and you have a set with exactly that many vectors, you only need to check one of the two basis conditions (either spanning or linear independence). The other condition will automatically be satisfied. This is because violating one condition would lead to a contradiction with our earlier theorems, particularly Theorem 4.5.2.

EXAMPLE 5: Bases by Inspection for \(R^{2}\)

Explain why the vectors \(\mathbf{v}_{1} = (-3,7)\) and \(\mathbf{v}_{2} = (5,5)\) form a basis for \(R^{2}\).

Solution (a):

1. We know \(\dim (R^{2}) = 2\).
2. The set \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}\}\) has exactly 2 vectors.
3. We check for linear independence: Neither \(\mathbf{v}_{1}\) nor \(\mathbf{v}_{2}\) is a scalar multiple of the other. \(k(-3,7) = (5,5)\) implies \(-3k=5\) (\(k=-5/3\)) and \(7k=5\) (\(k=5/7\)), which is a contradiction. Therefore, \(S\) is linearly independent.
4. By Theorem 4.5.4, since \(S\) is a linearly independent set with \(n=2\) vectors in an \(n=2\) dimensional space, \(S\) must also span \(R^{2}\) and thus forms a basis for \(R^{2}\).

Let’s apply Theorem 4.5.4. We know that \(R^2\) is 2-dimensional. We are given two vectors. All we need to do is check if they are linearly independent (or if they span \(R^2\)). A quick check reveals that \(\mathbf{v}_1\) and \(\mathbf{v}_2\) are not scalar multiples of each other, meaning they are linearly independent. Since we have exactly 2 linearly independent vectors in a 2-dimensional space, Theorem 4.5.4 guarantees they form a basis.

Interactive Example: Basis by Inspection in \(R^2\)

Let’s visualize \(\mathbf{v}_{1} = (-3,7)\) and \(\mathbf{v}_{2} = (5,5)\) and confirm linear independence.

Here’s a visual and computational check. We plot \(\mathbf{v}_1\) and \(\mathbf{v}_2\). Visually, they don’t lie on the same line, suggesting linear independence. Computationally, we form a matrix with these vectors and calculate its determinant. A non-zero determinant confirms linear independence. Since the determinant is 50, which is non-zero, the vectors are linearly independent. Given that \(R^2\) is 2-dimensional, these two vectors form a basis.

EXAMPLE 5: Bases by Inspection for \(R^{3}\)

Explain why \(\mathbf{v}_{1} = (2,0, - 1)\), \(\mathbf{v}_{2} = (4,0,7)\), and \(\mathbf{v}_{3} = (-1,1,4)\) form a basis for \(R^{3}\).

Solution (b):

1. We know \(\dim (R^{3}) = 3\).
2. The set \(S = \{\mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{3}\}\) has exactly 3 vectors.
3. We check for linear independence:
   • \(\mathbf{v}_{1}\) and \(\mathbf{v}_{2}\) are in the \(xz\)-plane (their \(y\)-components are 0). They are linearly independent since neither is a scalar multiple of the other.
   • \(\mathbf{v}_{3}\) has a non-zero \(y\)-component (1). This means \(\mathbf{v}_{3}\) is outside the \(xz\)-plane, and thus outside the \(\operatorname{span}(\{\mathbf{v}_{1}, \mathbf{v}_{2}\})\).
   • By the Plus/Minus Theorem (4.5.3a), adding \(\mathbf{v}_{3}\) to the linearly independent set \(\{\mathbf{v}_{1}, \mathbf{v}_{2}\}\) results in a linearly independent set \(\{\mathbf{v}_{1}, \mathbf{v}_{2}, \mathbf{v}_{3}\}\).
4. By Theorem 4.5.4, since \(S\) is a linearly independent set with \(n=3\) vectors in an \(n=3\) dimensional space, \(S\) must also span \(R^{3}\) and thus forms a basis for \(R^{3}\).

For \(R^3\), which is 3-dimensional, we have three vectors. We can use a similar logic. First, \(\mathbf{v}_1\) and \(\mathbf{v}_2\) are linearly independent and lie in the xz-plane. Now, \(\mathbf{v}_3\) has a non-zero y-component, meaning it’s not in the plane spanned by \(\mathbf{v}_1\) and \(\mathbf{v}_2\). By the Plus/Minus Theorem, adding \(\mathbf{v}_3\) to the set makes the full set \(\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}\) linearly independent. Since we have 3 linearly independent vectors in a 3-dimensional space, they form a basis.

THEOREM 4.5.5: Building and Reducing Bases

This theorem provides methods for constructing bases from spanning sets or linearly independent sets.

THEOREM 4.5.5

Let \(S\) be a finite set of vectors in a finite-dimensional vector space \(V\).

1. If \(S\) spans \(V\) but is not a basis for \(V\), then \(S\) can be reduced to a basis for \(V\) by removing appropriate vectors from \(S\).

2. If \(S\) is a linearly independent set that is not already a basis for \(V\), then \(S\) can be enlarged to a basis for \(V\) by inserting appropriate vectors into \(S\).

These processes are fundamental in many algorithms, such as Gram-Schmidt orthogonalization.

Theorem 4.5.5 is a constructive theorem. Part (a) tells us that if you have a set that spans the entire space but is redundant (not a basis), you can trim the fat by removing linearly dependent vectors until you arrive at a basis. Part (b) says if you have a linearly independent set that doesn’t quite span the whole space (not a basis), you can add vectors that are outside its span until you complete it into a basis. These principles are used in various algorithms, for example, in finding a basis for a subspace or in orthogonalization processes important for signal processing.

THEOREM 4.5.6: Dimension of Subspaces

The dimension of a subspace is always less than or equal to the dimension of the parent space.

THEOREM 4.5.6

If \(W\) is a subspace of a finite-dimensional vector space \(V\), then:

1. \(W\) is finite-dimensional.
2. \(\dim (W) \leq \dim (V)\).
3. \(W = V\) if and only if \(\dim (W) = \dim (V)\).

Figure 4.5.2: Geometric relationship between subspaces of \(R^3\).

Finally, Theorem 4.5.6 provides important properties about subspaces. Part (a) states that any subspace of a finite-dimensional space is itself finite-dimensional. Part (b) confirms our intuition that a subspace cannot be “larger” than the space it’s contained within; its dimension must be less than or equal to the parent space’s dimension. And part (c) is particularly insightful: if a subspace has the same dimension as its parent space, then it must actually be the parent space. This is very useful for proving equality of spaces. For instance, in \(R^3\), a 2-dimensional subspace is a plane through the origin, and a 1-dimensional subspace is a line through the origin, as illustrated in the figure.

Summary

• Dimension Definition: The number of vectors in any basis for a finite-dimensional vector space.
• Key Property (Theorem 4.5.1): All bases for a given vector space have the same number of vectors.
• Relationship (Theorem 4.5.2): Sets with too many vectors are dependent; sets with too few cannot span the space.
• Constructing Bases (Theorem 4.5.3 & 4.5.5): Methods for adding vectors to linearly independent sets or removing vectors from spanning sets to form a basis.
• Basis by Inspection (Theorem 4.5.4): If dimension is known, only one condition (spanning or independence) needs to be checked.
• Subspace Dimensions (Theorem 4.5.6): Subspaces are finite-dimensional, and their dimension is less than or equal to the parent space’s dimension.

In this section, we’ve formalized the concept of dimension, moving beyond intuition to a precise mathematical definition based on the number of vectors in a basis. We explored several fundamental theorems that govern the relationships between linear independence, spanning sets, and dimension. These concepts are absolutely critical for understanding the structure of vector spaces and are applied extensively in various ECE fields, from understanding signal spaces to solving complex system equations.