Artificial Intelligence vs Machine Learning vs Deep Learning

Activation Function

Relu?

#Means max(x,0)

def relu(x):     
    if (element < 0)
        Replace with 0
    else
        Keep element as it is

def naive_relu(x):                      #x is 2D Tensor
    assert len(x.shape) == 2

    x = x.copy()                        #copy to avoid changing input
    for i in range(x.shape[0]):         #x.shape[0]=2
        for j in range(x.shape[1]):     #x.shape[1]=3
            x[i, j] = max(x[i, j], 0)
return x            

a = np.array([              #Dimension=2. Shape(2,3)
            [0, 1, -2],
            [4, 5, -6],
        ])
b = naive_relu(a)
print(b)
[[0 1 0]
 [4 5 0]]

        

Adding activation function to a layer introduces non-linearity into the model, allowing it to learn more complex relationships between the input and output data

Non-linear activation functions such as ReLU, Sigmoid, and Tanh can help the model to better fit the training data and make more accurate predictions on new data.

Conda

Miniconda is the recommended approach for installing TensorFlow with GPU support

It creates a separate environment to avoid changing any installed software in your system. This is also the easiest way to install the required software especially for the GPU setup.

Optimizers

optimizer is an algorithm used to adjust the parameters of a model in order to minimize the error or loss function

from keras.optimizers import RMSprop
optimizer = RMSprop(learning_rate=0.001, rho=0.9)
                    

from keras.optimizers import SGD
optimizer = SGD(learning_rate=0.01, momentum=0.9)
                

Orchestration

Monitoring and handling ML pipelines
Steps in Orchestration?   1. Workflow Definition: Defining the sequence of steps (tasks)
  2. Automation: Automatically triggering and running the pipeline
  3. Resource Management: managing the underlying computational resources (CPUs, GPUs)
  4. Monitoring and Logging: Tracking the execution status, performance metrics

Bias/Sampling Bias

We should use a training data set that is representative of the cases we want model to predict.

if the sample is too small, you will have sampling noise (i.e., nonrepresentative data as a result of chance), but even very large samples can be nonrepresentative if the sampling method is flawed. This is called sampling bias.

CNTK

This is Microsoft Cognitive Toolkit (CNTK) backend, plugged with keras.

Fitting

Overfitting

Means that the model performs well on the training data, but it does not generalize well(ie produces good results on real world/unseen data), This is because model memorizes the exact relationship in the sample data including noise or minor details.
Overfitting Example on DecisionTreeRegressor

Underfitting

Does not produces good results on traning data and bad results on new data as well
Underfitting Example on DecisionTreeRegressor

Large language model

Llama (Large Language Model Meta AI)

Llama Models

Layer / class or Function

Layer processes input data(tensor) and produces an output(tensor) in specific format. Neural network is created by cascading multiple layers.
Types of Layers:

E. Generative AI

Loss Function

In Keras, a loss function is used during the training of a neural network. It measures the difference between the model's predictions and the actual target values. The goal of training is to minimize this loss

            Loss function = (Actual O/P) - (Expected output)
        

from keras.utils import to_categorical
train_labels = [0, 1, 2, 0, 1]
train_labels_one_hot = to_categorical(train_labels) # Convert to one-hot encoding
print(train_labels_one_hot)

array([[1., 0., 0.],                //represents 0
       [0., 1., 0.],                #1
       [0., 0., 1.],                #2
       [1., 0., 0.],                #0
       [0., 1., 0.]], dtype=float32)#1

        

Matplotlib

Metrics

Metrics to monitor during training and testing

Tensor = n-D Matrix

Types of Tensors

| 1 2 3 |
| 4 5 6 |
                

[
    | 1 2 3 |
    | 4 5 6 |,

    | 1 2 3 |
    | 4 5 6 |
]
                

            import numpy as np
            ########## 2-D Tensor ###########
            b = np.array(
                [
                    [0, 1, 2, 3],
                    [4, 5, 6, 7],
                    [8, 9, 10, 11],
                ]
            )
            print("Dimension/Ndim:", b.ndim)        # 2         //2d array
            print("Shape:", b.shape)                # (3, 4)    //(row,col)
            
            ########## 3-D Tensor, Packing 2-D matrices ###########
            c = np.array(
                [
                    [
                        [0, 1, 2],
                        [4, 5, 6],
                        [8, 9, 10],
                    ],
                    [
                        [10, 11, 12],
                        [14, 15, 16],
                        [18, 19, 110],
                    ]
                ]
            )
            print("Dimension/Ndim:", c.ndim)    # 3         //3d array
            print("Shape:", c.shape)            # (2,3,3)   //(2=arrays, 3=row, 3=col)

            ####### Operations ############
                #all,row,col
            d = c[:, 2:, 2:]            # Select all elements 2nd(row), 2nd(col) onwards.
            print(d)                    #   [[[ 10]]  [[110]]]
            

Tensor Operations

Add vector(dimension=1) + matrix(dimension=2)

|1 2 3| + | 1 2 3 | = |2 4 6|
          | 4 5 6 |   |5 7 9|

import numpy as np
def naive_add_matrix_and_vector(x, y):
    assert len(x.shape) == 2    #Matrix
    assert len(y.shape) == 1    #vector
    assert x.shape[1] == y.shape[0]

    x = x.copy()
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            x[i, j] += y[j]
    return x

x = np.array(
        [
            [1,2,3],
            [4,5,6]
        ]
)
y = np.array([1,2,3])
z = naive_add_matrix_and_vector(x,y)
print(z)
'''
[[2,4,6]
[5,7,9]]
'''
            

Dot Product (.)

[1,2,3].[2,3,4] = 1*2+2*3+3*4 = 20.0

import numpy as np
def naive_vector_dot(x, y):
    z = 0.                  #float
    for i in range(x.shape[0]):
        z += x[i] * y[i]
    return z

x = np.array([1,2,3])
y = np.array([2,3,4])
z = naive_vector_dot(x, y)
print(z)        #20.0    
                        

| 1 2 3 | . | 1 2 3 | = |1x1 + 2x2 + 3x3 , 1x4 + 2x5 + 3x6| 
            | 4 5 6 | = |13.0, 32.0|
import numpy as np
def naive_matrix_vector_dot(x, y):
    assert len(x.shape) == 2    #matrix
    assert len(y.shape) == 1    #vector
    assert x.shape[1] == y.shape[0]
    z = np.zeros(x.shape[0])
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            z[i] += x[i, j] * y[j]
    return z

y = np.array([1,2,3])       #y.shape = (3)
x = np.array(               #x.shape = (2,3) 
                [
                    [1,2,3],
                    [4,5,6]
                ]
            )
print(naive_matrix_vector_dot(x,y))
[14. 32.]
                        

Tensor Reshaping

Shape(3,2) to (6,1). (2,3)

>>> x = np.array([[0, 1],
                  [2, 3],
                  [4, 5]])
>>> print(x.shape)      #See above how shape(3,2)
(3, 2)
>>> x = x.reshape((6, 1))
>>> x
array([[ 0],
       [ 1],
       [ 2],
       [ 3],
       [ 4],
       [ 5]])
>>> x = x.reshape((2, 3))
>>> x
array([[ 0, 1, 2],
       [ 3, 4, 5]])
        

Transpose

>>> x = np.zeros((300, 20))
>>> x = np.transpose(x)
>>> print(x.shape)
(20, 300)
        

Tensor Terms

Dimension/Axis   Array                        
    0           np.array(12)                     // Point does not have any dimension
    1           np.array([1,2])                     // 1x2. 1 Dimensional
    2           np.array([[5, 78, 2, 34, 0],       // 3x4. 2 Dimensional
                        [6, 79, 3, 35, 1],
                        [7, 80, 4, 36, 2]])
                    

Variance

Variance is the tendency to learn random things unrelated to the real signal