Artificial Intelligence, Machine Learning, Deep Learning

Activation Function

Relu?

#Means max(x,0)

def relu(x):     
    if (element < 0)
        Replace with 0
    else
        Keep element as it is

def naive_relu(x):                      #x is 2D Tensor
    assert len(x.shape) == 2

    x = x.copy()                        #copy to avoid changing input
    for i in range(x.shape[0]):         #x.shape[0]=2
        for j in range(x.shape[1]):     #x.shape[1]=3
            x[i, j] = max(x[i, j], 0)
return x            

a = np.array([              #Dimension=2. Shape(2,3)
            [0, 1, -2],
            [4, 5, -6],
        ])
b = naive_relu(a)
print(b)
[[0 1 0]
 [4 5 0]]

        

Adding activation function to a layer introduces non-linearity into the model, allowing it to learn more complex relationships between the input and output data

Non-linear activation functions such as ReLU, Sigmoid, and Tanh can help the model to better fit the training data and make more accurate predictions on new data.

Conda

Miniconda is the recommended approach for installing TensorFlow with GPU support

It creates a separate environment to avoid changing any installed software in your system. This is also the easiest way to install the required software especially for the GPU setup.

Optimizers

optimizer is an algorithm used to adjust the parameters of a model in order to minimize the error or loss function

from keras.optimizers import RMSprop
optimizer = RMSprop(learning_rate=0.001, rho=0.9)
                    

from keras.optimizers import SGD
optimizer = SGD(learning_rate=0.01, momentum=0.9)
                

Overfitting

Means that the model performs well on the training data, but it does not generalize well(ie produces good results on real world/unseen data), because there is too of much uneccessary data(noise) in traning data.

Regularization: Constraining a model to make it simpler and reduce the risk of overfitting.

Bias/Sampling Bias

We should use a training data set that is representative of the cases we want model to predict.

if the sample is too small, you will have sampling noise (i.e., nonrepresentative data as a result of chance), but even very large samples can be nonrepresentative if the sampling method is flawed. This is called sampling bias.

CNTK

This is Microsoft Cognitive Toolkit (CNTK) backend, plugged with keras.

Keras

Library(in Python) which provides functions/APIs to build deep-learning models. Different backends can be plugged with keras

            Keras
            Tensorflow / Theano / CNTK
             CUDA           BLAS,Eigen
             GPU            CPU
        

Large language model

Llama (Large Language Model Meta AI)

Llama Models

Layer / class or Function

Learning Types

A. Supervised Learning

The training data feed to the algorithm includes the desired solutions(called labels). Supervised learning algorithms:

Types of Supervised learning

1. Classification (give yes/no)

Types of Classification models

2. Regression: (give % or numeric value)

B. Unsupervised Learning

Dataset does not have labels. ML has to estimate whether it's correct/incorrect and create its own rules

Types of unsupervised learning

1. Clustering

2. Visualization and dimensionality reduction

3. Association rule learning

C. Semisupervised Learning

D. Reinforcement learning

E. Generative AI

Loss Function

In Keras, a loss function is used during the training of a neural network. It measures the difference between the model's predictions and the actual target values. The goal of training is to minimize this loss

            Loss function = (Actual O/P) - (Expected output)
        

from keras.utils import to_categorical
train_labels = [0, 1, 2, 0, 1]
train_labels_one_hot = to_categorical(train_labels) # Convert to one-hot encoding
print(train_labels_one_hot)

array([[1., 0., 0.],                //represents 0
       [0., 1., 0.],                #1
       [0., 0., 1.],                #2
       [1., 0., 0.],                #0
       [0., 1., 0.]], dtype=float32)#1

        

Matplotlib

Metrics

Metrics to monitor during training and testing

Neuron / Node / Function or class

A neuron is the basic unit within a layer. It takes input, performs a computation, and produces an output.

Each neuron has weight, bias, activation function

class Neuron:
    def __init__(self, num_inputs):
        self.weights = initialize_weights(num_inputs)
        self.bias = initialize_bias()
        self.activation_function = relu

    def forward(self, input_data):
        # Compute the weighted sum of inputs
        weighted_sum = sum(weight * input_value for weight, input_value in zip(self.weights, input_data)) + self.bias
        
        # Apply the activation function
        output = self.activation_function(weighted_sum)
        
        return output        
        

Neural Network

Tensor = n-D Matrix

Types of Tensors

| 1 2 3 |
| 4 5 6 |
                

[
    | 1 2 3 |
    | 4 5 6 |,

    | 1 2 3 |
    | 4 5 6 |
]
                

            import numpy as np
            ########## 2-D Tensor ###########
            b = np.array(
                [
                    [0, 1, 2, 3],
                    [4, 5, 6, 7],
                    [8, 9, 10, 11],
                ]
            )
            print("Dimension/Ndim:", b.ndim)        # 2         //2d array
            print("Shape:", b.shape)                # (3, 4)    //(row,col)
            
            ########## 3-D Tensor, Packing 2-D matrices ###########
            c = np.array(
                [
                    [
                        [0, 1, 2],
                        [4, 5, 6],
                        [8, 9, 10],
                    ],
                    [
                        [10, 11, 12],
                        [14, 15, 16],
                        [18, 19, 110],
                    ]
                ]
            )
            print("Dimension/Ndim:", c.ndim)    # 3         //3d array
            print("Shape:", c.shape)            # (2,3,3)   //(2=arrays, 3=row, 3=col)

            ####### Operations ############
                #all,row,col
            d = c[:, 2:, 2:]            # Select all elements 2nd(row), 2nd(col) onwards.
            print(d)                    #   [[[ 10]]  [[110]]]
            

Tensor Operations

Add vector(dimension=1) + matrix(dimension=2)

|1 2 3| + | 1 2 3 | = |2 4 6|
          | 4 5 6 |   |5 7 9|

import numpy as np
def naive_add_matrix_and_vector(x, y):
    assert len(x.shape) == 2    #Matrix
    assert len(y.shape) == 1    #vector
    assert x.shape[1] == y.shape[0]

    x = x.copy()
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            x[i, j] += y[j]
    return x

x = np.array(
        [
            [1,2,3],
            [4,5,6]
        ]
)
y = np.array([1,2,3])
z = naive_add_matrix_and_vector(x,y)
print(z)
'''
[[2,4,6]
[5,7,9]]
'''
            

Dot Product (.)

[1,2,3].[2,3,4] = 1*2+2*3+3*4 = 20.0

import numpy as np
def naive_vector_dot(x, y):
    z = 0.                  #float
    for i in range(x.shape[0]):
        z += x[i] * y[i]
    return z

x = np.array([1,2,3])
y = np.array([2,3,4])
z = naive_vector_dot(x, y)
print(z)        #20.0    
                        

| 1 2 3 | . | 1 2 3 | = |1x1 + 2x2 + 3x3 , 1x4 + 2x5 + 3x6| 
            | 4 5 6 | = |13.0, 32.0|
import numpy as np
def naive_matrix_vector_dot(x, y):
    assert len(x.shape) == 2    #matrix
    assert len(y.shape) == 1    #vector
    assert x.shape[1] == y.shape[0]
    z = np.zeros(x.shape[0])
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            z[i] += x[i, j] * y[j]
    return z

y = np.array([1,2,3])       #y.shape = (3)
x = np.array(               #x.shape = (2,3) 
                [
                    [1,2,3],
                    [4,5,6]
                ]
            )
print(naive_matrix_vector_dot(x,y))
[14. 32.]
                        

Tensor Reshaping

Shape(3,2) to (6,1). (2,3)

>>> x = np.array([[0, 1],
                  [2, 3],
                  [4, 5]])
>>> print(x.shape)      #See above how shape(3,2)
(3, 2)
>>> x = x.reshape((6, 1))
>>> x
array([[ 0],
       [ 1],
       [ 2],
       [ 3],
       [ 4],
       [ 5]])
>>> x = x.reshape((2, 3))
>>> x
array([[ 0, 1, 2],
       [ 3, 4, 5]])
        

Transpose

>>> x = np.zeros((300, 20))
>>> x = np.transpose(x)
>>> print(x.shape)
(20, 300)
        

Tensor Terms

Dimension/Axis   Array                        
    0           np.array(12)                     // Point does not have any dimension
    1           np.array([1,2])                     // 1x2. 1 Dimensional
    2           np.array([[5, 78, 2, 34, 0],       // 3x4. 2 Dimensional
                        [6, 79, 3, 35, 1],
                        [7, 80, 4, 36, 2]])
                    

Tensorflow

This is ML Open source library(EXPOSING APIs) for numerical computation and large-scale ML supports CPUs & GPUs.

Python Front-end APIs & backend written in c++ for high performance.

            //Install conda https://docs.conda.io/projects/miniconda/en/latest/
            C:\Users\amitk\source\repos\Python> mkdir venv_ml1
            C:\Users\amitk\source\repos\Python> cd venv_ml1
            C:\Users\amitk\source\repos\Python\venv_ml1>"c:\Users\amitk\miniconda3\Scripts\activate" venv_ml1
            //Env is created here: C:\Users\amitk\miniconda3\envs
            (venv_ml1) C:\Users\amitk\source\repos\Python\venv_ml1>
            (venv_ml1) C:\Users\amitk\source\repos\Python\venv_ml1>"c:\Users\amitk\miniconda3\condabin\deactivate.bat"  //deactivate
            (venv_ml1) C:\Users\amitk\source\repos\Python\venv_ml1>pip install tensorflow
            Downloading tensorflow-2.6.2-cp36-cp36m-win_amd64.whl (423.3 MB)
        

Underfitting

Does not produces good results on traning data.

Variance

Variance is the tendency to learn random things unrelated to the real signal