Kubernetes how to get the CPU and memory usage

ZnJvbSBrdWJlcm5ldGVzIGltcG9ydCBjbGllbnQsIGNvbmZpZwoKIyBMb2FkIEt1YmVybmV0ZXMgY29uZmlndXJhdGlvbiBmcm9tIGRlZmF1bHQgbG9jYXRpb24gb3IgcHJvdmlkZSB5b3VyIG93biBrdWJlY29uZmlnIGZpbGUKY29uZmlnLmxvYWRfa3ViZV9jb25maWcoKQoKIyBDcmVhdGUgS3ViZXJuZXRlcyBBUEkgY2xpZW50CmFwaV9jbGllbnQgPSBjbGllbnQuQXBpQ2xpZW50KCkKCiMgUmV0cmlldmUgbGlzdCBvZiBub2RlcyBpbiB0aGUgY2x1c3Rlcgp2MSA9IGNsaWVudC5Db3JlVjFBcGkoYXBpX2NsaWVudCkKbm9kZV9saXN0ID0gdjEubGlzdF9ub2RlKCkuaXRlbXMKCiMgSXRlcmF0ZSBvdmVyIGVhY2ggbm9kZSBhbmQgcmV0cmlldmUgc3lzdGVtIHJlc291cmNlIHVzYWdlCmZvciBub2RlIGluIG5vZGVfbGlzdDoKICAgIG5vZGVfbmFtZSA9IG5vZGUubWV0YWRhdGEubmFtZQogICAgcHJpbnQoZiJOb2RlOiB7bm9kZV9uYW1lfSIpCgogICAgIyBSZXRyaWV2ZSBub2RlIG1ldHJpY3MKICAgIG1ldHJpY3NfYXBpID0gY2xpZW50LkN1c3RvbU9iamVjdHNBcGkoYXBpX2NsaWVudCkKICAgIG1ldHJpY3MgPSBtZXRyaWNzX2FwaS5saXN0X2NsdXN0ZXJfY3VzdG9tX29iamVjdCgibWV0cmljcy5rOHMuaW8iLCAidjFiZXRhMSIsICJub2RlcyIsICJub2RlIikKCiAgICAjIEZpbmQgdGhlIG1ldHJpY3MgZm9yIHRoZSBjdXJyZW50IG5vZGUKICAgIGZvciBtZXRyaWMgaW4gbWV0cmljc1siaXRlbXMiXToKICAgICAgICBpZiBtZXRyaWNbIm1ldGFkYXRhIl1bIm5hbWUiXSA9PSBub2RlX25hbWU6CiAgICAgICAgICAgIGNwdV91c2FnZSA9IG1ldHJpY1sidXNhZ2UiXVsiY3B1Il0KICAgICAgICAgICAgbWVtb3J5X3VzYWdlID0gbWV0cmljWyJ1c2FnZSJdWyJtZW1vcnkiXQogICAgICAgICAgICBwcmludChmIkNQVSBVc2FnZToge2NwdV91c2FnZX0iKQogICAgICAgICAgICBwcmludChmIk1lbW9yeSBVc2FnZToge21lbW9yeV91c2FnZX0iKQoKICAgIHByaW50KCItLS0tLS0tLS0tLS0tLS0tLS0tLSIp

This code snippet assumes you have the Kubernetes Python client library (kubernetes) installed. If you haven't installed it, you can use pip install kubernetes to install it.

The code first loads the Kubernetes configuration using config.load_kube_config(). This assumes that you have configured your Kubernetes credentials, or it uses the default configuration if available. Alternatively, you can provide the path to your kubeconfig file using config.load_kube_config(config_file='path/to/kubeconfig').

Then, it creates a Kubernetes API client using client.ApiClient(). This client will be used to interact with the Kubernetes API.

Next, it retrieves the list of nodes in the cluster using the list_node() method from the CoreV1Api class.

The code then iterates over each node and retrieves the system resource usage for each node. It uses the CustomObjectsApi class to query the metrics API and retrieve the node metrics. The metrics are stored in the metrics variable.

Within the loop, it searches for the metrics corresponding to the current node using the node name. It retrieves the CPU and memory usage from the metrics data.

Finally, it prints the node name, CPU usage, and memory usage for each node.

Please note that this code assumes that the metrics server is installed in your Kubernetes cluster and available at the metrics.k8s.io API group. If your cluster does not have the metrics server installed or uses a different metrics provider, you may need to adjust the code accordingly.

Also, ensure that you have the necessary permissions to access the metrics server and retrieve the required metrics.

Time Series analysis

 RFC

aW1wb3J0IHBhbmRhcyBhcyBwZApmcm9tIHNrbGVhcm4uZW5zZW1ibGUgaW1wb3J0IFJhbmRvbUZvcmVzdENsYXNzaWZpZXIKZnJvbSBza2xlYXJuLm1vZGVsX3NlbGVjdGlvbiBpbXBvcnQgdHJhaW5fdGVzdF9zcGxpdApmcm9tIHNrbGVhcm4ubWV0cmljcyBpbXBvcnQgYWNjdXJhY3lfc2NvcmUKCiMgUmVhZCBsb2cgZmlsZSBpbnRvIGEgcGFuZGFzIERhdGFGcmFtZQpkZiA9IHBkLnJlYWRfY3N2KCdsb2dmaWxlLmNzdicsIHBhcnNlX2RhdGVzPVsndGltZXN0YW1wJ10pCgojIFNldCB0aW1lc3RhbXAgYXMgdGhlIGluZGV4CmRmLnNldF9pbmRleCgndGltZXN0YW1wJywgaW5wbGFjZT1UcnVlKQoKIyBTZWxlY3QgdGhlIHJlbGV2YW50IGRlcGVuZGVudCB2YXJpYWJsZXMgYW5kIHRhcmdldCB2YXJpYWJsZSBmb3IgbW9kZWxpbmcKIyBSZXBsYWNlICd2YXIxJywgJ3ZhcjInLCAndmFyMycgd2l0aCB5b3VyIG93biB2YXJpYWJsZSBuYW1lcwojICd0YXJnZXQnIHNob3VsZCByZXByZXNlbnQgd2hldGhlciB0aGUgc2VydmVyIHdlbnQgZG93biAoMSkgb3Igbm90ICgwKQpkYXRhID0gZGZbWyd2YXIxJywgJ3ZhcjInLCAndmFyMycsICd0YXJnZXQnXV0KCiMgU3BsaXQgdGhlIGRhdGEgaW50byBmZWF0dXJlcyAoWCkgYW5kIHRhcmdldCAoeSkKWCA9IGRhdGFbWyd2YXIxJywgJ3ZhcjInLCAndmFyMyddXQp5ID0gZGF0YVsndGFyZ2V0J10KCiMgU3BsaXQgdGhlIGRhdGEgaW50byB0cmFpbiBhbmQgdGVzdCBzZXRzClhfdHJhaW4sIFhfdGVzdCwgeV90cmFpbiwgeV90ZXN0ID0gdHJhaW5fdGVzdF9zcGxpdChYLCB5LCB0ZXN0X3NpemU9MC4yLCByYW5kb21fc3RhdGU9NDIpCgojIENyZWF0ZSBhbmQgZml0IHRoZSBSYW5kb20gRm9yZXN0IENsYXNzaWZpZXIKbW9kZWwgPSBSYW5kb21Gb3Jlc3RDbGFzc2lmaWVyKCkKbW9kZWwuZml0KFhfdHJhaW4sIHlfdHJhaW4pCgojIFBlcmZvcm0gcHJlZGljdGlvbnMgb24gdGhlIHRlc3Qgc2V0CnByZWRpY3Rpb25zID0gbW9kZWwucHJlZGljdChYX3Rlc3QpCgojIENhbGN1bGF0ZSBhY2N1cmFjeQphY2N1cmFjeSA9IGFjY3VyYWN5X3Njb3JlKHlfdGVzdCwgcHJlZGljdGlvbnMpCnByaW50KGYnQWNjdXJhY3k6IHthY2N1cmFjeX0nKQoKIyBZb3UgY2FuIGFsc28gdXNlIHRoZSB0cmFpbmVkIG1vZGVsIHRvIHByZWRpY3Qgb24gbmV3IGRhdGEKIyBuZXdfZGF0YSA9IHBkLkRhdGFGcmFtZSh7J3ZhcjEnOiBbdmFsMV0sICd2YXIyJzogW3ZhbDJdLCAndmFyMyc6IFt2YWwzXX0pCiMgbmV3X3ByZWRpY3Rpb24gPSBtb2RlbC5wcmVkaWN0KG5ld19kYXRhKQ==

TSA Vector Auto Regression 

aW1wb3J0IHBhbmRhcyBhcyBwZApmcm9tIHN0YXRzbW9kZWxzLnRzYS52ZWN0b3JfYXIudmFyX21vZGVsIGltcG9ydCBWQVIKZnJvbSBza2xlYXJuLm1ldHJpY3MgaW1wb3J0IG1lYW5fc3F1YXJlZF9lcnJvcgppbXBvcnQgbWF0cGxvdGxpYi5weXBsb3QgYXMgcGx0CgojIFJlYWQgbG9nIGZpbGUgaW50byBhIHBhbmRhcyBEYXRhRnJhbWUKZGYgPSBwZC5yZWFkX2NzdignbG9nZmlsZS5jc3YnLCBwYXJzZV9kYXRlcz1bJ3RpbWVzdGFtcCddKQoKIyBTZXQgdGltZXN0YW1wIGFzIHRoZSBpbmRleApkZi5zZXRfaW5kZXgoJ3RpbWVzdGFtcCcsIGlucGxhY2U9VHJ1ZSkKCiMgU2VsZWN0IHRoZSByZWxldmFudCBkZXBlbmRlbnQgdmFyaWFibGVzIGZvciBtb2RlbGluZwpkYXRhID0gZGZbWyd2YXIxJywgJ3ZhcjInLCAndmFyMyddXSAgIyBSZXBsYWNlICd2YXIxJywgJ3ZhcjInLCAndmFyMycgd2l0aCB5b3VyIG93biB2YXJpYWJsZSBuYW1lcwoKIyBTcGxpdCB0aGUgZGF0YSBpbnRvIHRyYWluIGFuZCB0ZXN0IHNldHMKdHJhaW5fc2l6ZSA9IGludCgwLjggKiBsZW4oZGF0YSkpCnRyYWluX2RhdGEsIHRlc3RfZGF0YSA9IGRhdGFbOnRyYWluX3NpemVdLCBkYXRhW3RyYWluX3NpemU6XQoKIyBDcmVhdGUgYW5kIGZpdCB0aGUgVkFSIG1vZGVsCm1vZGVsID0gVkFSKHRyYWluX2RhdGEpCm1vZGVsX2ZpdCA9IG1vZGVsLmZpdCgpCgojIFBlcmZvcm0gcHJlZGljdGlvbnMgb24gdGhlIHRlc3Qgc2V0CmxhZ19vcmRlciA9IG1vZGVsX2ZpdC5rX2FyCnByZWRpY3Rpb25zID0gbW9kZWxfZml0LmZvcmVjYXN0KHRlc3RfZGF0YS52YWx1ZXMsIHN0ZXBzPWxlbih0ZXN0X2RhdGEpKQoKIyBDYWxjdWxhdGUgcm9vdCBtZWFuIHNxdWFyZWQgZXJyb3IgKFJNU0UpCnJtc2UgPSBtZWFuX3NxdWFyZWRfZXJyb3IodGVzdF9kYXRhLCBwcmVkaWN0aW9ucywgc3F1YXJlZD1GYWxzZSkKcHJpbnQoZidSTVNFOiB7cm1zZX0nKQoKIyBQbG90IHRoZSBhY3R1YWwgdmFsdWVzIGFuZCBwcmVkaWN0ZWQgdmFsdWVzCnBsdC5maWd1cmUoZmlnc2l6ZT0oMTIsIDYpKQpwbHQucGxvdCh0ZXN0X2RhdGEuaW5kZXgsIHRlc3RfZGF0YVsndmFyMSddLCBsYWJlbD0nQWN0dWFsJykKcGx0LnBsb3QodGVzdF9kYXRhLmluZGV4LCBwcmVkaWN0aW9uc1s6LCAwXSwgbGFiZWw9J1ByZWRpY3RlZCcpCnBsdC50aXRsZSgnVGltZSBTZXJpZXMgRm9yZWNhc3RpbmcnKQpwbHQueGxhYmVsKCdEYXRlJykKcGx0LnlsYWJlbCgnVmFyaWFibGUgMScpCnBsdC5sZWdlbmQoKQpwbHQuZ3JpZChUcnVlKQpwbHQuc2hvdygp

SARIMAX 

aW1wb3J0IHBhbmRhcyBhcyBwZApmcm9tIHN0YXRzbW9kZWxzLnRzYS5zdGF0ZXNwYWNlLnNhcmltYXggaW1wb3J0IFNBUklNQVgKaW1wb3J0IG1hdHBsb3RsaWIucHlwbG90IGFzIHBsdAoKIyBSZWFkIGxvZyBmaWxlIGludG8gYSBwYW5kYXMgRGF0YUZyYW1lCmRmID0gcGQucmVhZF9jc3YoJ2xvZ2ZpbGUuY3N2JywgcGFyc2VfZGF0ZXM9Wyd0aW1lc3RhbXAnXSkKCiMgU2V0IHRpbWVzdGFtcCBhcyB0aGUgaW5kZXgKZGYuc2V0X2luZGV4KCd0aW1lc3RhbXAnLCBpbnBsYWNlPVRydWUpCgojIFByZXBhcmUgdGhlIGRhdGEgZm9yIG1vZGVsaW5nCiMgQXNzdW1pbmcgdGhlICdjb3VudCcgY29sdW1uIHJlcHJlc2VudHMgdGhlIHN5c3RlbSBlcnJvciBjb3VudApkYXRhID0gZGZbWydjb3VudCddXS5jb3B5KCkKCiMgU3BsaXQgdGhlIGRhdGEgaW50byB0cmFpbiBhbmQgdGVzdCBzZXRzCnRyYWluX3NpemUgPSBpbnQoMC44ICogbGVuKGRhdGEpKQp0cmFpbl9kYXRhLCB0ZXN0X2RhdGEgPSBkYXRhWzp0cmFpbl9zaXplXSwgZGF0YVt0cmFpbl9zaXplOl0KCiMgQ3JlYXRlIGFuZCBmaXQgdGhlIFNBUklNQVggbW9kZWwKbW9kZWwgPSBTQVJJTUFYKHRyYWluX2RhdGEsIG9yZGVyPSgxLCAxLCAxKSwgc2Vhc29uYWxfb3JkZXI9KDEsIDEsIDEsIDI0KSkKbW9kZWxfZml0ID0gbW9kZWwuZml0KCkKCiMgUGVyZm9ybSBwcmVkaWN0aW9ucyBvbiB0aGUgdGVzdCBzZXQKcHJlZGljdGlvbnMgPSBtb2RlbF9maXQucHJlZGljdChzdGFydD1sZW4odHJhaW5fZGF0YSksIGVuZD1sZW4odHJhaW5fZGF0YSkgKyBsZW4odGVzdF9kYXRhKSAtIDEpCgojIFBsb3QgdGhlIGFjdHVhbCB2YWx1ZXMgYW5kIHByZWRpY3RlZCB2YWx1ZXMKcGx0LmZpZ3VyZShmaWdzaXplPSgxMiwgNikpCnBsdC5wbG90KGRhdGEuaW5kZXhbdHJhaW5fc2l6ZTpdLCB0ZXN0X2RhdGEsIGxhYmVsPSdBY3R1YWwnKQpwbHQucGxvdChkYXRhLmluZGV4W3RyYWluX3NpemU6XSwgcHJlZGljdGlvbnMsIGxhYmVsPSdQcmVkaWN0ZWQnKQpwbHQudGl0bGUoJ1N5c3RlbSBFcnJvciBQcmVkaWN0aW9uJykKcGx0LnhsYWJlbCgnRGF0ZScpCnBsdC55bGFiZWwoJ0Vycm9yIENvdW50JykKcGx0LmxlZ2VuZCgpCnBsdC5ncmlkKFRydWUpCnBsdC5zaG93KCk=

Dictionary with depth and width

 Nice code for creating dict with depth and width

def create_nested_dict(depth, width):

    if depth == 0:

        return {}

    nested_dict = {}

    nested_dict['level'] = depth

    nested_dict['data'] = {}

    

    for i in range(width):

        nested_dict['data'][f'key_{i}'] = create_nested_dict(depth - 1, width)

    

    return nested_dict

# Set the desired depth and width of the nested dictionary

depth = 3

width = 2

# Create the nested dictionary

nested_dict = create_nested_dict(depth, width)

# Print the nested dictionary

import pprint

pprint.pprint(nested_dict)

MS Research: What are some of the best alogorithms for log file anslysis

ARIMA (AutoRegressive Integrated Moving Average): ARIMA models are a class of linear models that can capture trends and seasonality in time series data. They can be used for forecasting log file metrics such as request volume, error rates, and response times.

LSTM (Long Short-Term Memory): LSTM is a type of recurrent neural network that is well-suited for modeling sequences of data with long-term dependencies. They can be used for forecasting log file metrics such as network traffic, resource utilization, and user behavior.

Prophet: Prophet is a forecasting library developed by Facebook that is designed for time series data with strong seasonal patterns. It can be used for forecasting log file metrics such as web traffic, page views, and user activity.

Holt-Winters: Holt-Winters is a triple exponential smoothing method that can be used for forecasting time series data with trends and seasonality. It can be used for forecasting log file metrics such as system performance, application usage, and user engagement.

VAR (Vector Autoregression): VAR is a multivariate time series model that can capture dependencies between multiple variables. It can be used for forecasting log file metrics such as resource allocation, system utilization, and user interactions.

references:

MS Research - Timeseries for Log analysis

This one is a good research paper 

Our goal is to develop models for the analysis of searchers’

behaviors over time and investigate if time series analysis is a valid method for predicting

relationships between searcher actions. Time series analysis is a method often used to

understand the underlying characteristics of temporal data in order to make forecasts. In

this study, we used a Web search engine transactional log and time series analysis to investigate users’ actions. We conducted our analysis in two phases. In the initial phase, we

employed a basic analysis and found that 10% of searchers clicked on sponsored links.

However, from 22:00 to 24:00, searchers almost exclusively clicked on the organic links,

with almost no clicks on sponsored links. In the second and more extensive phase, we used

a one-step prediction time series analysis method along with a transfer function method.

The period rarely affects navigational and transactional queries, while rates for transactional queries vary during different periods. Our results show that the average length of

a searcher session is approximately 2.9 interactions and that this average is consistent

across time periods. Most importantly, our findings shows that searchers who submit

the shortest queries (i.e., in number of terms) click on highest ranked results. We discuss

implications, including predictive value, and future research

references:

https://faculty.ist.psu.edu/jjansen/academic/jansen_time_series_analysis.pdf

What is TensorFlow Gradient Tape

The most useful application of Gradient Tap is when you design a custom layer in your keras model for example--or equivalently designing a custom training loop for your model.

If you have a custom layer, you can define exactly how the operations occur within that layer, including the gradients that are computed and also calculating the amount of loss that is accumulated.

So Gradient tape will just give you direct access to the individual gradients that are in the layer.

Here is an example from Aurelien Geron's 2nd edition book on Tensorflow.

Say you have a function you want as your activation.

 def f(w1, w2):

     return 3 * w1 ** 2 + 2 * w1 * w2

Now if you want to take derivatives of this function with respec to w1 and w2:

w1, w2 = tf.Variable(5.), tf.Variable(3.)

with tf.GradientTape() as tape:

    z = f(w1, w2)

gradients = tape.gradient(z, [w1, w2])

So the optimizer will calculate the gradient and give you access to those values. Then you can double them, square them, triple them, etc., whatever you like. Whatever you choose to do, then you can add those adjusted gradients to the loss calculation for the backpropagation step, etc.

references

Docker install specific version on linux

sudo dnf --refresh update

sudo dnf upgrade

sudo dnf install yum-utils

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

sudo dnf install docker-ce docker-ce-cli containerd.io docker-compose-plugin

If a specific version to be installed, below can be done 

yum list docker-ce --showduplicates | sort -r

to remove older versions, below needs to be done 

sudo yum remove -y docker-ce docker-ce-cli

[cloud_user@info2c ~]$ yum list docker-ce --showduplicates | sort -r

docker-ce.x86_64                3:20.10.2-3.el8                 docker-ce-stable

docker-ce.x86_64                3:20.10.1-3.el8                 docker-ce-stable

docker-ce.x86_64                3:20.10.0-3.el8                 docker-ce-stable

docker-ce.x86_64                3:19.03.14-3.el8                docker-ce-stable

docker-ce.x86_64                3:19.03.13-3.el8                docker-ce-stable

now to install the specific version

sudo yum install docker-ce-20.10.2 docker-ce-cli-20.10.2 containerd.io

references:

https://ostechnix.com/install-docker-almalinux-centos-rocky-linux/