Linear Algebra with NumPy

One of the major pisions of mathematics is algebra, and linear algebra in particular, focuses on linear equations and mapping linear spaces, namely vector spaces. When we create a linear map between vector spaces, we are actually creating a data structure called a matrix. The main usage of linear algebra is to solve simultaneous linear equations, but it can also be used for approximations for non-linear systems. Imagine a complex model or system that you are trying to understand, think of it as a non-linear model. In such cases, you can reduce the complex, non-linear characteristics of the problem into simultaneous linear equations, and you can solve them with the help of linear algebra.

In computer science, linear algebra is heavily used in machine learning (ML) applications. In ML applications, you deal with high-dimensional arrays, which can easily be turned into linear equations where you can analyze the interaction of features in a given space. Imagine a case where you are working on an image recognition project and your task is to detect a tumor in the brain from MRI images. Technically, your algorithm should act like a doctor, where it scans the given input and detects the tumor in the brain. A doctor has the advantage of being able to spot anomalies; the human brain has been evolving through thousands of years to interpret visual input. Without much effort, a human can capture anomalies intuitively. However, for an algorithm to perform a similar task, you should think about this process in as much detail as possible to understand how you can formally express it so that the machines can understand.

First, you should think about how MRI data is stored in a computer, which processes only 0s and 1s. The computer actually stores pixel intensities in structures, called matrices. In other words, you will convert an MRI as a vector of dimensions, N2, where each element consists of pixel values. If this MRI has a 512 x 512 dimension, each pixel will be one point in 262,144 in pixels. Therefore, any computational manipulation that you will do in this Matrix would most likely use Linear Algebra principles. If this example is not enough to demonstrate the importance of Linear Algebra in ML, then let's look at a popular example in deep learning. In a nutshell, deep learning is an algorithm that uses neural network structure to learn the desired output (label) by continuously updating the weights of neuron connections between layers. A graphical representation of a simple deep learning algorithm is as follows:

Neural networks store weights between layers and bias values in matrices. As these are the parameters that you try to tune in to your deep learning model in order to minimize your loss function, you continuously make computations and update them. In general, ML models require heavy calculations and need to be trained for big datasets to provide efficient results. This is why linear algebra is a fundamental part of ML.

In this chapter, we will use numpy library but note that most linear algebra functions are also imported by scipy and they are more properly belong to it. Ideally, in most cases, you import both of these libraries and perform computations. One important feature of scipy is to have fully-featured versions of the linear algebra modules. We highly encourage you to review scipy documentation and practice using same operations with scipy throughout this chapter. Here's the link for linear algebra module of scipy: https://docs.scipy.org/doc/scipy/reference/linalg.html.

In this chapter, we will cover the following topics:

• Vector and matrix mathematics 
• What's an eigenvalue and how do we compute it?
• Computing the norm and determinant
• Solving linear equations
• Computing gradient