Artificial Intelligence vs Machine Learning vs Deep Learning

Activation Function

Relu: Relu is operation carried on 2D Tensor. if value is less than 0, Take 0 else take the original value

#Means max(x,0)

def relu(x):     
    if (element < 0)
        Replace with 0
    else
        Keep element as it is

def naive_relu(x):                      #x is 2D Tensor
    assert len(x.shape) == 2

    x = x.copy()                        #copy to avoid changing input
    for i in range(x.shape[0]):         #x.shape[0]=2
        for j in range(x.shape[1]):     #x.shape[1]=3
            x[i, j] = max(x[i, j], 0)
return x            

a = np.array([              #Dimension=2. Shape(2,3)
            [0, 1, -2],
            [4, 5, -6],
        ])
b = naive_relu(a)
print(b)
[[0 1 0]
 [4 5 0]]

        

Adding activation function to a layer introduces non-linearity into the model, allowing it to learn more complex relationships between the input and output data
Non-linear activation functions such as ReLU, Sigmoid, and Tanh can help the model to better fit the training data and make more accurate predictions on new data.

Axis

Pandas

In Pandas Dataframe: axis can have 2 values only:
  0=index The vertical axis. Operations move down the rows.
  1=columns The horizontal axis. Operations move across the columns.

Conda

Miniconda is the recommended approach for installing TensorFlow with GPU support
It creates a separate environment to avoid changing any installed software in your system. This is also the easiest way to install the required software especially for the GPU setup.

Optimizers

optimizer is an algorithm used to adjust the parameters of a model in order to minimize the error or loss function

from keras.optimizers import RMSprop
optimizer = RMSprop(learning_rate=0.001, rho=0.9)
                    

from keras.optimizers import SGD
optimizer = SGD(learning_rate=0.01, momentum=0.9)
                

Orchestration

Monitoring and handling ML pipelines
Steps in Orchestration?
  1. Workflow Definition: Defining the sequence of steps (tasks)
  2. Automation: Automatically triggering and running the pipeline
  3. Resource Management: managing the underlying computational resources (CPUs, GPUs)
  4. Monitoring and Logging: Tracking the execution status, performance metrics

Bias/Sampling Bias

We should use a training data set that is representative of the cases we want model to predict.
if the sample is too small, you will have sampling noise (i.e., nonrepresentative data as a result of chance), but even very large samples can be nonrepresentative if the sampling method is flawed. This is called sampling bias.

CNTK

This is Microsoft Cognitive Toolkit (CNTK) backend, plugged with keras.

Embedding Models

Embdedding models are used in RAG pipelines. They take unstructured data(eg: text) and breaks into vectors/tensors

Fitting

Overfitting

Means that the model performs well on the training data, but it does not generalize well(ie produces good results on real world/unseen data), This is because model memorizes the exact relationship in the sample data including noise or minor details.
Overfitting Example on DecisionTreeRegressor

Underfitting

Does not produces good results on traning data and bad results on new data as well
Underfitting Example on DecisionTreeRegressor

Large language model

This is a computational model that can do natural language processing tasks such as classification.
LLM is not an executable(exe), or shared library(so file). It doesn’t contain machine instructions (x86_64 or ARM)
An LLM/AI model is a massive multidimensional array of numbers (a static data file), coupled with a specific file structure metadata layout. These numbers are called weights and biases.
When a model "learns" during training, it isn't writing code. It is adjusting millions or billions of floating-point numbers until those numbers mathematically represent patterns in data. LLMs learns from by self-supervised and semi-supervised training.
Examples:
- Opensource Models: Llama, Gemma, Phi, Llama 3.2, Qwen 2.5 3B
- Closedsource Models: gpt(openai), Claude (Anthropic).

Llama(Meta's LLM) Models

LLM Model's Efficiency

1. Parameters: The "1B", "3B", or "8B" stands for Billion Parameters(the number of internal connections inside the AI's brain). More parameters usually mean a smarter model, but it requires more memory.
2. Quantization: Floating point numbers for weights/calculations.
3. Speed (Tokens per Second): Tokens are fractions of words (100 tokens $\approx$ 75 words).An efficient model running on a good GPU will hit 40 to 100+ tokens per second, printing text faster than you can read it. If it drops below 10-15 tokens/sec, it feels sluggish.
4. Resource Utilization(Fitting in VRAM): Does the model fit entirely inside your Graphics Card's memory (VRAM)? If it fits in VRAM, it runs at lightning speed. If it overflows into your system RAM, performance drops off a cliff.

Layer / class or Function

Layer processes input data(tensor) and produces an output(tensor) in specific format. Neural network is created by cascading multiple layers.
Types of Layers:

E. Generative AI

Class of models that creates content from user input. For example, generative AI can create unique images, music etc
Example:
Text-to-text
Text-to-image
Text-to-video
Text-to-code
Text-to-speech
Image and text-to-image

Loss Function

In Keras, a loss function is used during the training of a neural network. It measures the difference between the model's predictions and the actual target values. The goal of training is to minimize this loss

 Loss function = (Actual O/P) - (Expected output) 

Function to convert labels into a one-hot encoded format.
One-hot encoding is converting categorical labels into a binary matrix (1s and 0s)

from keras.utils import to_categorical
train_labels = [0, 1, 2, 0, 1]
train_labels_one_hot = to_categorical(train_labels) # Convert to one-hot encoding
print(train_labels_one_hot)

array([[1., 0., 0.],                //represents 0
       [0., 1., 0.],                #1
       [0., 0., 1.],                #2
       [1., 0., 0.],                #0
       [0., 1., 0.]], dtype=float32)#1

        

Matplotlib

Popular plotting library for Python that provides a variety of high-quality 2D and 3D plots and visualizations.
Matplotlib.pyplot is a collection of functions that make Matplotlib work like MATLAB, allowing you to create plots, charts
imshow(digit, cmap=plt.cm.binary) is used to display images in digit array, cmap=colormap for mapping the data values to colors in the plot. plt.cm.binary = black and white colors.

Metrics

Metrics to monitor during training and testing

Tensor = n-D Matrix

This is matrix(as in maths). Multi-dimensional numpy arrays used to store numbers during computation.

vector? This is 1-D matrix(ie array).

Types of Tensors

| 1 2 3 |
| 4 5 6 |
                

[
    | 1 2 3 |
    | 4 5 6 |,

    | 1 2 3 |
    | 4 5 6 |
]
                

            import numpy as np
            ########## 2-D Tensor ###########
            b = np.array(
                [
                    [0, 1, 2, 3],
                    [4, 5, 6, 7],
                    [8, 9, 10, 11],
                ]
            )
            print("Dimension/Ndim:", b.ndim)        # 2         //2d array
            print("Shape:", b.shape)                # (3, 4)    //(row,col)
            
            ########## 3-D Tensor, Packing 2-D matrices ###########
            c = np.array(
                [
                    [
                        [0, 1, 2],
                        [4, 5, 6],
                        [8, 9, 10],
                    ],
                    [
                        [10, 11, 12],
                        [14, 15, 16],
                        [18, 19, 110],
                    ]
                ]
            )
            print("Dimension/Ndim:", c.ndim)    # 3         //3d array
            print("Shape:", c.shape)            # (2,3,3)   //(2=arrays, 3=row, 3=col)

            ####### Operations ############
                #all,row,col
            d = c[:, 2:, 2:]            # Select all elements 2nd(row), 2nd(col) onwards.
            print(d)                    #   [[[ 10]]  [[110]]]
            

Tensor Operations

Add vector(dimension=1) + matrix(dimension=2)

|1 2 3| + | 1 2 3 | = |2 4 6|
          | 4 5 6 |   |5 7 9|

import numpy as np
def naive_add_matrix_and_vector(x, y):
    assert len(x.shape) == 2    #Matrix
    assert len(y.shape) == 1    #vector
    assert x.shape[1] == y.shape[0]

    x = x.copy()
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            x[i, j] += y[j]
    return x

x = np.array(
        [
            [1,2,3],
            [4,5,6]
        ]
)
y = np.array([1,2,3])
z = naive_add_matrix_and_vector(x,y)
print(z)
'''
[[2,4,6]
[5,7,9]]
'''
            

Dot Product (.)

[1,2,3].[2,3,4] = 1*2+2*3+3*4 = 20.0

import numpy as np
def naive_vector_dot(x, y):
    z = 0.                  #float
    for i in range(x.shape[0]):
        z += x[i] * y[i]
    return z

x = np.array([1,2,3])
y = np.array([2,3,4])
z = naive_vector_dot(x, y)
print(z)        #20.0    
                        

| 1 2 3 | . | 1 2 3 | = |1x1 + 2x2 + 3x3 , 1x4 + 2x5 + 3x6| 
            | 4 5 6 | = |13.0, 32.0|
import numpy as np
def naive_matrix_vector_dot(x, y):
    assert len(x.shape) == 2    #matrix
    assert len(y.shape) == 1    #vector
    assert x.shape[1] == y.shape[0]
    z = np.zeros(x.shape[0])
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            z[i] += x[i, j] * y[j]
    return z

y = np.array([1,2,3])       #y.shape = (3)
x = np.array(               #x.shape = (2,3) 
                [
                    [1,2,3],
                    [4,5,6]
                ]
            )
print(naive_matrix_vector_dot(x,y))
[14. 32.]
                        

Tensor Reshaping

Shape(3,2) to (6,1). (2,3)

>>> x = np.array([[0, 1],
                  [2, 3],
                  [4, 5]])
>>> print(x.shape)      #See above how shape(3,2)
(3, 2)
>>> x = x.reshape((6, 1))
>>> x
array([[ 0],
       [ 1],
       [ 2],
       [ 3],
       [ 4],
       [ 5]])
>>> x = x.reshape((2, 3))
>>> x
array([[ 0, 1, 2],
       [ 3, 4, 5]])
        

Transpose

>>> x = np.zeros((300, 20))
>>> x = np.transpose(x)
>>> print(x.shape)
(20, 300)
        

Tensor Terms

Dimension/Axis   Array                        
    0           np.array(12)                     // Point does not have any dimension
    1           np.array([1,2])                     // 1x2. 1 Dimensional
    2           np.array([[5, 78, 2, 34, 0],       // 3x4. 2 Dimensional
                        [6, 79, 3, 35, 1],
                        [7, 80, 4, 36, 2]])
                    

Variance

Variance is the tendency to learn random things unrelated to the real signal