Sunday, September 11, 2022

Python Major modules required for AI/ML

pip install opencv-python  => For OpenCV 

pip install keras

pip install opencv-python

pip install tensorflow

pip install PIL


Some of the import statements are

from imageio import imread 
import random as rn
from keras import backend as K
import tensorflow as tf
import matplotlib.pyplot as plt
import cv2

 

Python how to print GPU info


pip install gputil

pip install tabulate


However this below was returning empty on an Macbook M1 Pro Chip


 import GPUtil

from tabulate import tabulate

print("="*40, "GPU Details", "="*40)

gpus = GPUtil.getGPUs()

print("GPU list ",gpus)

list_gpus = []

for gpu in gpus:

    # get the GPU id

    gpu_id = gpu.id

    # name of GPU

    gpu_name = gpu.name

    # get % percentage of GPU usage of that GPU

    gpu_load = f"{gpu.load*100}%"

    # get free memory in MB format

    gpu_free_memory = f"{gpu.memoryFree}MB"

    # get used memory

    gpu_used_memory = f"{gpu.memoryUsed}MB"

    # get total memory

    gpu_total_memory = f"{gpu.memoryTotal}MB"

    # get GPU temperature in Celsius

    gpu_temperature = f"{gpu.temperature} °C"

    gpu_uuid = gpu.uuid

    list_gpus.append((

        gpu_id, gpu_name, gpu_load, gpu_free_memory, gpu_used_memory,

        gpu_total_memory, gpu_temperature, gpu_uuid

    ))

print(tabulate(list_gpus, headers=("id", "name", "load", "free memory", "used memory", "total memory", "temperature", "uuid")))

Python how to print the System info

 import platform,socket,re,uuid,json,psutil,logging


def getSystemInfo():

    try:

        info={}

        info['platform']=platform.system()

        info['platform-release']=platform.release()

        info['platform-version']=platform.version()

        info['architecture']=platform.machine()

        info['hostname']=socket.gethostname()

        info['ip-address']=socket.gethostbyname(socket.gethostname())

        info['mac-address']=':'.join(re.findall('..', '%012x' % uuid.getnode()))

        info['processor']=platform.processor()

        info['ram']=str(round(psutil.virtual_memory().total / (1024.0 **3)))+" GB"

        return json.dumps(info)

    except Exception as e:

        logging.exception(e)


json.loads(getSystemInfo())


Docker postgres container how to export and import data from

Below two commands help to do that 

docker exec -u postgres test_postgres pg_dump -Cc | xz > test-backup-$(date -u +%Y-%m-%d).sql.xz

xz -dc test-backup-2022-09-11.sql.xz | docker exec -i -u postgres test_postgres psql 


Below is some explanation of the switch options 

-u postgres 

We want to run the command as the postgres user because the docker exec command defaults to using the root user and the root user does not have access to the database.

test-postgres

This is the name of the Docker container running PostgreSQL. If you created the container with a different name, substitute it here.

pg_dump

pg_dump is the PostgreSQL database backup utility. It converts a database to an SQL script. It can also convert to some other formats, but we aren’t going to use those right now.

-Cc

Equivalent to –create –clean.

–create tells pg_dump to include tables, views, and functions in the backup, not just the data contained in the tables.

–clean tells pg_dump to start the SQL script by dropping the data that is currently in the database. This makes it easier to restore in one step.

| xz

We run the SQL script through a compression program (in this case, xz) to reduce the amount of storage space taken by the backup and to reduce the time it takes to transfer the backup over the network. This is optional, and other commands can be used in place of xz, such as gzip and bzip2. To get even better compression, the -9 and/or -e options can be specified. -9 makes xz use much more memory, and -e makes xz use much more CPU power. However, the default compression level should be good enough in nearly every case.

> proget-backup-$(date -u +%Y-%m-%d).sql.xz

The compressed SQL script is currently being written on the standard output, so we redirect it to a file with a name like proget-backup-2022-09-11.sql.xz. This will be placed in the current directory when you run the command.

xz -dc proget-backup-2022-09-11.sql.xz |

Because we compressed the SQL script in the previous command, we need to decompress it before we can restore the backup. -dc is equivalent to –decompress –stdout.

–decompress tells xz that we want to decompress the file, not compress it again.

–stdout tells xz that it should write the contents of the file to the standard output and not delete the .xz file. Without this, xz will write the output to a file named proget-backup-2022-09-11.sql and delete the compressed version.

gzip and bzip2 both use the same meaning for -dc as xz.

-i

This tells Docker to keep the standard input open so the SQL script can be sent to psql.

psql

This is the PostgreSQL interactive SQL command line. In this case, we’re using it to run the SQL script containing the database backup.

–set ON_ERROR_STOP=on

Tells psql to stop executing the restore if an error occurs.

–single-transaction

Tells psql to run the entire restore in one transaction so that any problem that causes it to stop doesn’t leave the database in an inconsistent state.


references:
https://inedo.com/support/kb/1145/accessing-a-postgresql-database-in-a-docker-container

Docker postgres environment variables

 

POSTGRES_PASSWORD

POSTGRES_USER

PGDATA

POSTGRES_DB

POSTGRES_INITDB_ARGS


Environment variables can be overridden using below three methods depending on your case.


Run Image: If running docker image directly, use below to include environment variable in docker run with -e K=V. Like mentioned here https://docs.docker.com/engine/reference/run/


Dockerfile: If you need to specify the environment variable in Dockerfile, specify as mentioned below


ENV POSTGRES_PASSWORD=secrect

ENV POSTGRES_USER=postgres



Docker how to shell into bash

 the general format is 

docker exec -it (container) (command)


docker exec -t -i container_name /bin/bash

docker exec -ti container_name /bin/bash

docker exec -ti container_name sh


Use opener (https://github.com/artemkaxboy/docker-opener):

requires to add alias in your environment opener wordpress

works anywhere docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock artemkaxboy/opener wordpress

Instead of wordpress you can use name or id or image-name of any container you want to connect

references


Docker copying from to host

 sudo docker cp container-id:/path/filename.txt ~/Desktop/filename.txt

Obtain the name or id of the Docker container

Issue the docker cp command and reference the container name or id

The first parameter of the docker copy command is the path to the file inside the container

The second parameter of the docker copy command is the location to save the file on the host

Edit and use the file copied from inside the container to your host machine

references:

https://www.theserverside.com/blog/Coffee-Talk-Java-News-Stories-and-Opinions/How-to-copy-files-from-a-Docker-container-to-a-host-machine