MS Research: What are some of the best alogorithms for log file anslysis

ARIMA (AutoRegressive Integrated Moving Average): ARIMA models are a class of linear models that can capture trends and seasonality in time series data. They can be used for forecasting log file metrics such as request volume, error rates, and response times.

LSTM (Long Short-Term Memory): LSTM is a type of recurrent neural network that is well-suited for modeling sequences of data with long-term dependencies. They can be used for forecasting log file metrics such as network traffic, resource utilization, and user behavior.

Prophet: Prophet is a forecasting library developed by Facebook that is designed for time series data with strong seasonal patterns. It can be used for forecasting log file metrics such as web traffic, page views, and user activity.

Holt-Winters: Holt-Winters is a triple exponential smoothing method that can be used for forecasting time series data with trends and seasonality. It can be used for forecasting log file metrics such as system performance, application usage, and user engagement.

VAR (Vector Autoregression): VAR is a multivariate time series model that can capture dependencies between multiple variables. It can be used for forecasting log file metrics such as resource allocation, system utilization, and user interactions.

references:

MS Research - Timeseries for Log analysis

This one is a good research paper 

Our goal is to develop models for the analysis of searchers’

behaviors over time and investigate if time series analysis is a valid method for predicting

relationships between searcher actions. Time series analysis is a method often used to

understand the underlying characteristics of temporal data in order to make forecasts. In

this study, we used a Web search engine transactional log and time series analysis to investigate users’ actions. We conducted our analysis in two phases. In the initial phase, we

employed a basic analysis and found that 10% of searchers clicked on sponsored links.

However, from 22:00 to 24:00, searchers almost exclusively clicked on the organic links,

with almost no clicks on sponsored links. In the second and more extensive phase, we used

a one-step prediction time series analysis method along with a transfer function method.

The period rarely affects navigational and transactional queries, while rates for transactional queries vary during different periods. Our results show that the average length of

a searcher session is approximately 2.9 interactions and that this average is consistent

across time periods. Most importantly, our findings shows that searchers who submit

the shortest queries (i.e., in number of terms) click on highest ranked results. We discuss

implications, including predictive value, and future research

references:

https://faculty.ist.psu.edu/jjansen/academic/jansen_time_series_analysis.pdf

What is TensorFlow Gradient Tape

The most useful application of Gradient Tap is when you design a custom layer in your keras model for example--or equivalently designing a custom training loop for your model.

If you have a custom layer, you can define exactly how the operations occur within that layer, including the gradients that are computed and also calculating the amount of loss that is accumulated.

So Gradient tape will just give you direct access to the individual gradients that are in the layer.

Here is an example from Aurelien Geron's 2nd edition book on Tensorflow.

Say you have a function you want as your activation.

 def f(w1, w2):

     return 3 * w1 ** 2 + 2 * w1 * w2

Now if you want to take derivatives of this function with respec to w1 and w2:

w1, w2 = tf.Variable(5.), tf.Variable(3.)

with tf.GradientTape() as tape:

    z = f(w1, w2)

gradients = tape.gradient(z, [w1, w2])

So the optimizer will calculate the gradient and give you access to those values. Then you can double them, square them, triple them, etc., whatever you like. Whatever you choose to do, then you can add those adjusted gradients to the loss calculation for the backpropagation step, etc.

references

Docker install specific version on linux

sudo dnf --refresh update

sudo dnf upgrade

sudo dnf install yum-utils

sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

sudo dnf install docker-ce docker-ce-cli containerd.io docker-compose-plugin

If a specific version to be installed, below can be done 

yum list docker-ce --showduplicates | sort -r

to remove older versions, below needs to be done 

sudo yum remove -y docker-ce docker-ce-cli

[cloud_user@info2c ~]$ yum list docker-ce --showduplicates | sort -r

docker-ce.x86_64                3:20.10.2-3.el8                 docker-ce-stable

docker-ce.x86_64                3:20.10.1-3.el8                 docker-ce-stable

docker-ce.x86_64                3:20.10.0-3.el8                 docker-ce-stable

docker-ce.x86_64                3:19.03.14-3.el8                docker-ce-stable

docker-ce.x86_64                3:19.03.13-3.el8                docker-ce-stable

now to install the specific version

sudo yum install docker-ce-20.10.2 docker-ce-cli-20.10.2 containerd.io

references:

https://ostechnix.com/install-docker-almalinux-centos-rocky-linux/

Python how to get the caller file name and line number

 import inspect

def info(msg):

    frm = inspect.stack()[1]

    print('frm is ',frm)

    mod = inspect.getmodule(frm[0])

    print(mod.__name__)

    

info('hello')

references:

https://stackoverflow.com/questions/1095543/get-name-of-calling-functions-module-in-python

What is double and single underscore meaning in Python

 While none of this is strictly enforced by python, the naming convention of a double underscore means "private", while a single underscore means "protected".

A double underscore is meant to protect subclasses from causing errors by using the same name. By namespacing them by class name, the defining class can be sure its variables will remain valid.

A single underscore means that subclasses are free to access and modify those same attributes, being shared with the super class.

Both forms suggest that any outside class should not be accessing these.

class A(object):

    __private = 'private'

    _protected = 'protected'

    def show(self):

        print self.__private 

        print self._protected

class B(A):

    __private = 'private 2'

    _protected = 'protected 2'

a = A()

a.show()

#private

#protected

b = B()

b.show()

#private

#protected 2

This example shows that even though class B defined a new __private, it does not affect the inherited show method, which still accesses the original superclass attribute. _protected is however modified and the superclass show method reflects this, because they are the same attribute.

references:

https://stackoverflow.com/questions/12117087/python-hide-methods-with

How to provide a custom Filter for the Python logging

# server.py
import uuid
import logging
import flask
from flask import Flask

def get_request_id():
    if getattr(flask.g, 'request_id', None):
        return flask.g.request_id

    new_uuid = uuid.uuid4().hex[:10]
    flask.g.request_id = new_uuid

    return new_uuid

class RequestIdFilter(logging.Filter):
    # This is a logging filter that makes the request ID available for use in
    # the logging format. Note that we're checking if we're in a request
    # context, as we may want to log things before Flask is fully loaded.
    def filter(self, record):
        record.req_id = get_request_id() if flask.has_request_context() else ''
        return True


logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# The StreamHandler responsible for displaying
# logs in the console.
sh = logging.StreamHandler()
sh.addFilter(RequestIdFilter())

# Note: the "req_id" param name must be the same as in
# RequestIdFilter.filter
log_formatter = logging.Formatter(
    fmt="%(module)s: %(asctime)s - %(levelname)s - ID: %(req_id)s - %(message).1000s"
)
sh.setFormatter(log_formatter)
logger.addHandler(sh)

app = Flask(__name__)


@app.route("/")
def hello():
    logger.info("Hello world!")
    logger.info("I am a log inside the /hello endpoint")
    return "Hello World!"


if __name__ == "__main__":
    app.run()

references:

https://stackoverflow.com/questions/69363431/add-session-id-or-some-unique-id-per-session-in-python-logs