Python docker SDK how to list all the containers

this can be done by giving all=True parameter like below 

client.containers.list(all=True) 

references:

https://stackoverflow.com/questions/54306722/python-docker-client-only-listing-running-containers

Python callback functions

def callbackFunc(s):

    print('Length of the text file is : ', s)

def printFileLength(path, callback):

    f = open(path, "r")

    length = len(f.read())

    f.close()

    callback(length)

if __name__ == '__main__':

    printFileLength("sample.txt", callbackFunc)

references:

 https://pythonexamples.org/python-callback-function/

React spread operator to operate on array

If you want to use the spread operator to add items to the beginning of an array, just use the spread operator after the new values. For example:

let a = [1, 2, 3, 4, 5];

let b = [0, ...a];

console.log(a);

console.log(b);

Additionally, you can also use .concat:

var a = [1, 2, 3, 4, 5];

var b = [0].concat(a);

console.log(a);

console.log(b);

references: 

https://stackoverflow.com/questions/42160400/react-redux-add-object-to-start-of-array-using-spread-operator

Flask How to send back JSON response

from flask import Flask,jsonify,request

  app =   Flask(__name__)

  

@app.route('/returnjson', methods = ['GET'])

def ReturnJSON():

    if(request.method == 'GET'):

        data = {

            "Modules" : 15,

            "Subject" : "Data Structures and Algorithms",

        }

  

        return jsonify(data)

  

if __name__=='__main__':

    app.run(debug=True) 

references:

https://www.geeksforgeeks.org/how-to-return-a-json-response-form-a-flask-api/

Flask App supporting all type of request methods

Below is how we need to do this 

from flask import Flask

from flask import request

app = Flask(__name__)

@app.route('/users/<user_id>', methods = ['GET', 'POST', 'DELETE'])
def user(user_id):
    if request.method == 'GET':
        """return the information for <user_id>"""
        .
        .
        .
    if request.method == 'POST':
        """modify/update the information for <user_id>"""
        # you can use <user_id>, which is a str but could
        # changed to be int or whatever you want, along
        # with your lxml knowledge to make the required
        # changes
        data = request.form # a multidict containing POST data
        .
        .
        .
    if request.method == 'DELETE':
        """delete user with ID <user_id>"""
        .
        .
        .
    else:
        # POST Error 405 Method Not Allowed
        .
        .
        

references:

https://stackoverflow.com/questions/22947905/flask-example-with-post

A simplest of simple Logistic regression example

This below example loads the data and does logistic regression and stores that in the pickle file and then does the prediction  

# Save Model Using Pickle

import pandas

from sklearn import model_selection

from sklearn.linear_model import LogisticRegression

import pickle

url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv"

names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']

dataframe = pandas.read_csv(url, names=names)

array = dataframe.values

X = array[:,0:8]

Y = array[:,8]

test_size = 0.33

seed = 7

X_train, X_test, Y_train, Y_test = model_selection.train_test_split(X, Y, test_size=test_size, random_state=seed)

# Fit the model on training set

model = LogisticRegression()

model.fit(X_train, Y_train)

# save the model to disk

filename = 'finalized_model.sav'

pickle.dump(model, open(filename, 'wb'))

 

# some time later...

 

# load the model from disk

loaded_model = pickle.load(open(filename, 'rb'))

result = loaded_model.score(X_test, Y_test)

print(result)

references:

https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/

How to install Scikit Learn and set it up

Better to use virtualenv 

pip install -U scikit-learn

In order to check your installation you can use

python -m pip show scikit-learn  # to see which version and where scikit-learn is installed

python -m pip freeze  # to see all packages installed in the active virtualenv

python -c "import sklearn; sklearn.show_versions()"

These are some of the outputs 

python -m pip show scikit-learn 

Name: scikit-learn

Version: 1.2.2

Summary: A set of python modules for machine learning and data mining

Home-page: http://scikit-learn.org

Author: None

Author-email: None

License: new BSD

Location: /Users/userrm/Documents/RR/projects/POCs/AIBot/BW2/server/venv/lib/python3.8/site-packages

Requires: threadpoolctl, scipy, numpy, joblib

Required-by: 

python -m pip freeze

bidict==0.22.1

click==8.1.3

Flask==2.2.3

Flask-SocketIO==5.3.3

importlib-metadata==6.1.0

itsdangerous==2.1.2

Jinja2==3.1.2

joblib==1.2.0

MarkupSafe==2.1.2

numpy==1.24.2

pandas==1.5.3

python-dateutil==2.8.2

python-engineio==4.4.0

python-socketio==5.8.0

pytz==2023.2

scikit-learn==1.2.2

scipy==1.10.1

six==1.16.0

threadpoolctl==3.1.0

Werkzeug==2.2.3

zipp==3.15.0

references:

https://scikit-learn.org/stable/install.html

sklearn convergence warning

Below warning what i was getting while executing sk learn logistic regression. 

ConvergenceWarning: lbfgs failed to converge (status=1):

STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:

    https://scikit-learn.org/stable/modules/preprocessing.html

Please also refer to the documentation for alternative solver options:

    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression

  n_iter_i = _check_optimize_result(

The warning means what it mainly says: Suggestions to try to make the solver (the algorithm) converges.

lbfgs stand for: "Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm". It is one of the solvers' algorithms provided by Scikit-Learn Library.

The term limited-memory simply means it stores only a few vectors that represent the gradients approximation implicitly.

It has better convergence on relatively small datasets.

But what is algorithm convergence?

In simple words. If the error of solving is ranging within very small range (i.e., it is almost not changing), then that means the algorithm reached the solution (not necessary to be the best solution as it might be stuck at what so-called "local Optima").

On the other hand, if the error is varying noticeably (even if the error is relatively small [like in your case the score was good], but rather the differences between the errors per iteration is greater than some tolerance) then we say the algorithm did not converge.

Now, you need to know that Scikit-Learn API sometimes provides the user the option to specify the maximum number of iterations the algorithm should take while it's searching for the solution in an iterative manner:

LogisticRegression(... solver='lbfgs', max_iter=100 ...)

As you can see, the default solver in LogisticRegression is 'lbfgs' and the maximum number of iterations is 100 by default.

Final words, please, however, note that increasing the maximum number of iterations does not necessarily guarantee convergence, but it certainly helps!

references:

https://stackoverflow.com/questions/62658215/convergencewarning-lbfgs-failed-to-converge-status-1-stop-total-no-of-iter

pip how to check if we are inside the virtual environment

pip -V is the option  

When not in the virual environment, it gives the path in the system 

(base) REKUMARM-M-WB8Y:server userrm$ pip -V

pip 20.2.4 from /opt/anaconda3/lib/python3.8/site-packages/pip (python 3.8)

now if we are inside the virual env, it gives the pip from the path of virtual env 

for e.g. like this below 

(base) REKUMARM-M-WB8Y:server userrm$ source venv/bin/activate

(venv) (base) REKUMARM-M-WB8Y:server userrm $ pip -V

pip 20.1.1 from /Users/userrm/Documents/CR/projects/POCs/BW2/server/venv/lib/python3.8/site-packages/pip (python 3.8)

references:

https://stackoverflow.com/questions/1871549/determine-if-python-is-running-inside-virtualenv#comment35880305_1883251 

pip Virtual Env Error

Below error was coming in when trying to create virtual env 

virtualenv botserver

-bash: /usr/local/bin/virtualenv: /usr/bin/python: bad interpreter: No such file or directory

None of these really worked, but then finally doing liek this below solved the issue 

python3 -m venv venv

source venv/bin/activate

references:

https://stackoverflow.com/questions/66069215/the-client-is-using-an-unsupported-version-of-the-socket-io-or-engine-io-protoco

Python flask Socket io Errors

 127.0.0.1 - - [25/Mar/2023 23:49:10] "GET / HTTP/1.1" 200 -

The client is using an unsupported version of the Socket.IO or Engine.IO protocols (further occurrences of this error will be logged with level INFO)

127.0.0.1 - - [25/Mar/2023 23:49:11] "GET /socket.io/?EIO=3&transport=polling&t=OSP_l7y HTTP/1.1" 400 -

This is because the index page was using 2.1.0 version of socket io causing this issue. 

Changing to like below solved the issue 

<!DOCTYPE html>

<html lang="en">

   <head>

      <meta charset="UTF-8">

      <title>SocketIO example</title>

      <script src="https://code.jquery.com/jquery-3.3.1.js"></script>

      <script src="//cdnjs.cloudflare.com/ajax/libs/socket.io/4.5.0/socket.io.js"></script>

      <script type="text/javascript">

         $(document).ready(function() {

           console.log('on ready function call ')

           // sending a connect request to the server.

           var socket = io.connect('http://127.0.0.1:5000');

           console.log('connection complete  ') 

           socket.on('after connect', function(msg) {

               console.log('After connect', msg);

               $('#log').append('<br>' + $('<div/>').text('Received: ' + msg.data).html());

          });        

         });

      </script>

   </head>

   <body>

    <h1>SocketIO Example</h1>

    <h2>Receive:</h2>

    <div id="log"></div>

   </body>

</html>

references: 

https://stackoverflow.com/questions/66069215/the-client-is-using-an-unsupported-version-of-the-socket-io-or-engine-io-protoco

HTML iframe get parent window location details

We need to use window.parent 

window.parent.location.href

references:

https://stackoverflow.com/questions/3420004/access-parent-url-from-iframe 

Implementing WebSocket with Flask And Python

Setup your virtual environment

python3 -m venv venv

Source (or activate) the virtual environment

source venv/bin/activate

First install Flask if you haven’t done so:

pip install flask

Install Flask-SocketIO with pip:

pip install flask_socketio

Python Flask Websocket

In the example below socket.io and the Flask module are used. socketio is an two-way event-based communication engine.

Events are triggered both by server and connected clients in socketio. If an event is detected, the related call back functions is called.

To implement event triggers or event callbacks in Flask is easy (The example below uses web sockets.)

Create a new file called app.py with this code:

from flask import Flask, render_template

from flask_socketio import SocketIO, emit

app = Flask(__name__)

socketio = SocketIO(app)

@app.route('/')

def index():

    return render_template('index.html')

@socketio.on('connect')

def test_connect():

    emit('after connect',  {'data':'Lets dance'})

if __name__ == '__main__':

    socketio.run(app)

then create a directory /templates/ with the file index.html. This uses JQuery and socket.io

<!DOCTYPE html>

<html lang="en">

   <head>

      <meta charset="UTF-8">

      <title>SocketIO example</title>

      <script src="https://code.jquery.com/jquery-3.3.1.js"></script>

      <script src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/2.1.1/socket.io.js"></script>

      <script type="text/javascript">

         $(document).ready(function() {

         

           // sending a connect request to the server.

           var socket = io.connect('http://localhost:5000');

         

           socket.on('after connect', function(msg) {

               console.log('After connect', msg);

               $('#log').append('<br>' + $('<div/>').text('Received: ' + msg.data).html());

          });        

         });

      </script>

   </head>

   <body>

    <h1>SocketIO Example</h1>

    <h2>Receive:</h2>

    <div id="log"></div>

   </body>

</html>

Start your app with:

python app.py

references:

https://pythonprogramminglanguage.com/python-flask-websocket/

What is CUDA

CUDA is a parallel computing platform and programming model developed by NVIDIA for general computing on its own GPUs (graphics processing units). CUDA enables developers to speed up compute-intensive applications by harnessing the power of GPUs for the parallelizable part of the computation.

While there have been other proposed APIs for GPUs, such as OpenCL, and there are competitive GPUs from other companies, such as AMD, the combination of CUDA and NVIDIA GPUs dominates several application areas, including deep learning, and is a foundation for some of the fastest computers in the world.

Graphics cards are arguably as old as the PC—that is, if you consider the 1981 IBM Monochrome Display Adapter a graphics card. By 1988, you could get a 16-bit 2D VGA Wonder card from ATI (the company eventually acquired by AMD). By 1996, you could buy a 3D graphics accelerator from 3dfx so that you could run the first-person shooter game Quake at full speed.

Google collab and Flask

 https://manivannan-ai.medium.com/rest-api-in-google-colab-f06cb01fadb0

How to view certificate in java key store

Manually check the cert using keytool

Check the chain using openSSL

1. Lets start with the manual check:

keytool -list -v -keystore my.certificate.chain.jks | grep -A 1 "Owner"

references:

https://blog.niklasottosson.com/misc/how-to-check-a-certificate-chain-in-a-jks/

Options for displaying sub html pages within another main

One is ofcourse iframe. Like below 

< iframe
    src="http://www.google.co.in"
    name="targetframe"
    allowTransparency="true"
    scrolling="no"
    frameborder="0"
>
< /iframe>

Another seems to be using jquery like below 

<html> 
  <head> 
    <script src="jquery.js"></script> 
    <script> 
    $(function(){
      $("#divId").load("b.html"); 
    });
    </script> 
  </head> 

  <body> 
     <div id="divId" style="display:none;"></div>
  </body> 
</html>

references:

https://stackoverflow.com/questions/12032664/load-a-html-page-within-another-html-page

Simple chat popup window like UI

This is a pretty simple code for displaying something within a single html page. 

references:

https://www.w3schools.com/howto/howto_js_popup_chat.asp

AI/ML How to refer a file from Google drive in a python notebook

from google.colab import files

src = list(files.upload().values())[0]

open('utils.py','wb').write(src)

one the file is uploaded, it get copied to the home folder and then can be imported like below 

import utils 

What does CountVectorizer do?

CountVectorizer is a great tool provided by the scikit-learn library in Python. It is used to transform a given text into a vector on the basis of the frequency (count) of each word that occurs in the entire text. This is helpful when we have multiple such texts, and we wish to convert each word in each text into vectors (for using in further text analysis). Let us consider a few sample texts from a document (each as a list element):

CountVectorizer creates a matrix in which each unique word is represented by a column of the matrix, and each text sample from the document is a row in the matrix. The value of each cell is nothing but the count of the word in that particular text sample.  This can be visualized as follows –

  at each four geek geeks geeksforgeeks help helps many one other two

document[0] 0 0 0 1 1 0 0 1 0 1 0 1

document[1] 0 0 1 0 2 0 1 0 0 0 0 1

document[2] 1 1 0 1 1 1 0 1 1 0 1 0

https://www.geeksforgeeks.org/using-countvectorizer-to-extracting-features-from-text/

What is Label Encoder & onehot encoder

In ML models we are often required to convert the categorical i.e text features to its numeric representation. The two most common ways to do this is to use Label Encoder or OneHot Encoder. 

Label Encoder:

Label Encoding in Python can be achieved using Sklearn Library. Sklearn provides a very efficient tool for encoding the levels of categorical features into numeric values. LabelEncoder encode labels with a value between 0 and n_classes-1 where n is the number of distinct labels. If a label repeats it assigns the same value to as assigned earlier.

The problem here is since there are different numbers in the same column, the model will misunderstand the data to be in some kind of order, 0 < 1 <2.

The model may derive a correlation like as the country number increases the population increases but this clearly may not be the scenario in some other data or the prediction set. To overcome this problem, we use One Hot Encoder.

One Hot Encoder:

Now, as we already discussed, depending on the data we have, we might run into situations where, after label encoding, we might confuse our model into thinking that a column has data with some kind of order or hierarchy when we clearly don’t have it. To avoid this, we ‘OneHotEncode’ that column.

What one hot encoding does is, it takes a column which has categorical data, which has been label encoded and then splits the column into multiple columns. The numbers are replaced by 1s and 0s, depending on which column has what value. In our example, we’ll get four new columns, one for each country — Japan, U.S, India, and China.

references:

https://towardsdatascience.com/choosing-the-right-encoding-method-label-vs-onehot-encoder-

What Is NetFlow?

NetFlow is a network protocol developed by Cisco for collecting IP traffic information and monitoring network flow. By analyzing NetFlow data, you can get a picture of network traffic flow and volume.

NetFlow is a one-way technology, so when the server responds to the initial client request, the process works in reverse and creates a new flow record. Using a NetFlow monitoring solution can allow you to monitor and analyze these flow records more efficiently and effectively for traffic within the network.

SolarWinds® NetFlow Traffic Analyzer (NTA) is a powerful and affordable NetFlow management solution with comprehensive monitoring tools designed to translate granular detail into easy-to-understand graphs and reports—helping you more clearly identify the largest resource drains your bandwidth.

references:

https://www.solarwinds.com/netflow-traffic-analyzer/use-cases/what-is-netflow

Solving lightgbm errors on Mac

Lightgbm OSError, Library not loaded

elf._handle = _dlopen(self._name, mode)

OSError: dlopen(/usr/local/lib/python2.7/site-packages/lightgbm/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib

Referenced from: /usr/local/lib/python2.7/site-packages/lightgbm/lib_lightgbm.so

Reason: image not found

There were a couple of solutions provided

brew install libomp

OR 

brew install cmake  

brew install gcc --without-multilib  

git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM  

mkdir build ; cd build  

cmake ..   

make -j  

cd ../python-packages  

sudo python setup.py install --precompile

OR 

conda install lightgbm

The one that worked for me was 

conda install lightgbm

 

References:

https://stackoverflow.com/questions/44937698/lightgbm-oserror-library-not-loaded

MLOps reference links

 

MLOps Architectures 

https://aws.amazon.com/blogs/machine-learning/build-mlops-workflows-with-amazon-sagemaker-projects-gitlab-and-gitlab-pipelines/

https://aws.amazon.com/blogs/apn/taming-machine-learning-on-aws-with-mlops-a-reference-architecture/ => Best one 

https://aws.amazon.com/blogs/machine-learning/train-machine-learning-models-using-amazon-keyspaces-as-a-data-source/ => Simple 

https://towardsdatascience.com/industrializing-an-ml-platform-with-amazon-sagemaker-studio-91b597802afe => Having Sagemaker inner details in diagram 

https://towardsdatascience.com/mlops-with-mlflow-and-amazon-sagemaker-pipelines-33e13d43f238 => This has MLFlow + Sagemaker 

https://aws.amazon.com/blogs/machine-learning/architect-and-build-the-full-machine-learning-lifecycle-with-amazon-sagemaker/

https://awstip.com/deep-dive-to-train-deploy-and-evaluate-a-model-in-amazon-sagemaker-3c30514d9d17 => Simple basic using AWS step by step

https://mlinproduction.com/sagemaker-architecture/ => Another tutorial 

https://aws.amazon.com/blogs/machine-learning/architect-and-build-the-full-machine-learning-lifecycle-with-amazon-sagemaker/ => Good one too 

https://aws.amazon.com/blogs/mt/setting-up-machine-learning-environments-aws/ => Various environments 

https://docs.aws.amazon.com/whitepapers/latest/build-secure-enterprise-ml-platform/automation-pipelines.html => Building a Secure MLOps Env

https://www.youtube.com/watch?v=Yz4NsQ4zl9g => From 17:00 - awesome 

https://medium.com/trigger-ai/mlops-in-practice-machine-learning-ml-model-deployment-patterns-part-1-ce7cb575feda 

https://towardsdatascience.com/mlops-in-practice-de-constructing-an-ml-solution-architecture-into-10-components-c55c88d8fc7a => 10 Elements required for MLOps 

Using Opensource tools 

https://aicurious.io/blog/2022-03-26-airflow-mlflow-or-kubeflow-for-mlops

https://medium.com/@amrkmrc/airflow-and-mlflow-integration-d93e193b8b74

https://mymlops.com/examples/airflow-mlflow

https://www.vantage-ai.com/en/blog/keeping-your-ml-model-in-shape-with-kafka-airflow-and-mlflow

Other readings 

https://www.kdnuggets.com/2022/10/top-10-mlops-tools-optimize-manage-machine-learning-lifecycle.html

https://dzone.com/articles/using-machine-learning-for-log-analysis-and-anomal => log analysis