Monday, June 28, 2021

What is a whl file ?

WHL file is a package saved in the Wheel format, which is the standard built-package format used for Python distributions. It contains all the files for a Python install and metadata, which includes the version of the wheel implementation and specification used to package it.


The Wheel format was introduced in PEP 427, which is a Python Enhancement Proposal authored by Daniel Holth and accepted in 2012. It was developed as a quicker and more reliable method of installing Python software than re-building from source code every time. WHL files only have to be moved to the correct location on the target system to be installed, whereas a source distribution requires a build step before installation.




References:

https://fileinfo.com/extension/whl#:~:text=A%20WHL%20file%20is%20a,specification%20used%20to%20package%20it.

What is importlib in python?

The purpose of the importlib package is two-fold. One is to provide the implementation of the import statement (and thus, by extension, the __import__() function) in Python source code. This provides an implementation of import which is portable to any Python interpreter. This also provides an implementation which is easier to comprehend than one implemented in a programming language other than Python.

Two, the components to implement import are exposed in this package, making it easier for users to create their own custom objects (known generically as an importer) to participate in the import process.



For relative imports you have to:

  • a) use relative name
  • b) provide anchor explicitly


importlib.import_module('.c', 'a.b')


Of course, you could also just do absolute import instead:



References

https://stackoverflow.com/questions/10675054/how-to-import-a-module-in-python-with-importlib-import-module

Saturday, June 26, 2021

Facebook Ad Debuggings

After doing the below, app was not showing test ads and neither the real ads. 

var bannerAdView: FBAdView!

 func loadBannerAd() {

        NSLog("loadBannerAd ENTRY")

        bannerAdView = FBAdView(placementID: "516263056231409_519678395889875", adSize: kFBAdSizeHeight50Banner, rootViewController: self)

        bannerAdView.backgroundColor = UIColor.red

        bannerAdView.frame = CGRect(x: 0.0, y: 20.0, width: self.view.bounds.size.width, height: 50.0)

        let testmode = FBAdSettings.isTestMode()

        print("testmode ",testmode)

        bannerAdView.delegate = self

        self.view.addSubview(bannerAdView)

        bannerAdView.loadAd()

    }

    func adView(adView: FBAdView, didFailWithError error: NSError) {

        NSLog("adview didFailWithError ")

    }

     

    func adViewDidClick(adView: FBAdView) {

        print("adViewDidClick tap on ad view")

        NSLog("adViewDidClick ")

    }

    

    func adViewDidLoad(adview:FBAdView){

        NSLog("adViewDidLoad ENTRY")

        NSLog("adViewDidLoad ")

    }

    

    func adView(didFailWithError error: NSError) {

        NSLog("Error loading ads ", error)

        print(error)

    }


running on the simulator, the testmode was giving  false. However, from the FBAdSettings simulator supposed to 


the main methods used for debugging were 

  let testDeviceHash = FBAdSettings.testDeviceHash()

  print("testDeviceHash HAsh is ",testDeviceHash)

  FBAdSettings.addTestDevice(testDeviceHash)



FBAdSettings.setLogLevel(FBAdLogLevel.debug)


This actually gives the error like below 


2021-06-27 10:55:33.061994+0530 TestDev[77253:1940584] [FBAudienceNetworkLog/FBAdProvider:164 thread:1 <error>] Ad request error: Error Domain=com.facebook.ads.sdk Code=1001 "No fill" UserInfo={NSLocalizedDescription=No fill, FBAdErrorDetailKey={

    msg = "No fill";

}}


And looking at the error code descriptions from the link in references, 


When testing your ad placements, Facebook will intentionally send a no-fill for about 20% of requests to allow you to test how your app handles the no-fill case.


And this below was an important point 


Test Mobile Apps

Once you have added a test user and added their test device, distribute your app to your test user's devices using your preferred distribution method.

Your test user must have the Facebook app installed on their device and login with the account you added as a test user. This is required to allow Facebook to serve your test user your ads.

Your test user can now trigger a test to see a production ad. See step 5 and 6 of the Add a Test Device section.

And below were the no fill reasons

  • Error 1001 - No Fill. May be due to a number of reasons including:
  • User not logged into Native Facebook App on Mobile Device
  • You have informed Facebook through the setAdvertiserTrackingEnabled flag to not deliver personalized ads via Audience Network.
  • The person did not give permission for Facebook to use their activity, that other apps and websites send to Facebook, to personalize the ads Facebook delivers to that person.
  • No Ad Inventory for current user
  • Your testing device must have the native Facebook application installed.
  • Your application should attempt to make another request after 30 seconds.

references:

https://developers.facebook.com/docs/reference/ios/current/class/FBAdSettings/

https://developers.facebook.com/docs/audience-network/setting-up/testing/platform#errors

https://developers.facebook.com/docs/audience-network/setting-up/test/checklist-errors/


Friday, June 25, 2021

What is Django Q

Django Q is a native Django task queue, scheduler and worker application using Python multiprocessing.



  • Multiprocessing worker pools
  • Asynchronous tasks
  • Scheduled, cron and repeated tasks
  • Signed and compressed packages
  • Failure and success database or cache
  • Result hooks, groups and chains
  • Django Admin integration
  • PaaS compatible with multiple instances
  • Multi cluster monitor
  • Redis, Disque, IronMQ, SQS, MongoDB or ORM
  • Rollbar and Sentry support


Django Q is tested with: Python 3.7 and 3.8, Django 2.2.x and 3.1.x


Below are the main components in Django Q


Signed Tasks


Tasks are first pickled and then signed using Django’s own django.core.signing module using the SECRET_KEY and cluster name as salt, before being sent to a message broker. This ensures that task packages on the broker can only be executed and read by clusters and django servers who share the same secret key and cluster name. If a package fails to unpack, it will be marked failed with the broker and discarded. Optionally the packages can be compressed before transport.


Broker

The broker collects task packages from the django instances and queues them for pick up by a cluster. If the broker supports message receipts, it will keep a copy of the tasks around until a cluster acknowledges the processing of the task. Otherwise it is put back in the queue after a timeout period. This ensure at-least-once delivery. Most failed deliveries will be the result of a worker or the cluster crashing before the task was saved.


Pusher

The pusher process continuously checks the broker for new task packages. It checks the signing and unpacks the task to the internal Task Queue. The amount of tasks in the Task Queue can be configured to control memory usage and minimize data loss in case of a failure.


Worker

A worker process pulls a task of the Task Queue and it sets a shared countdown timer with Sentinel indicating it is about to start work. The worker then tries to execute the task and afterwards the timer is reset and any results (including errors) are saved to the package. Irrespective of the failure or success of any of these steps, the package is then pushed onto the Result Queue.



Monitor

The result monitor checks the Result Queue for processed packages and saves both failed and successful packages to the Django database or cache backend. If the broker supports it, a delivery receipt is sent. In case the task was part of a chain, the next task is queued.


Sentinel

The sentinel spawns all process and then checks the health of all workers, including the pusher and the monitor. This includes checking timers on each worker for timeouts. In case of a sudden death or timeout, it will reincarnate the failing processes. When a stop signal is received, the sentinel will halt the pusher and instruct the workers and monitor to finish the remaining items. 


Timeouts

Before each task execution the worker sets a countdown timer on the sentinel and resets it again after execution. Meanwhile the sentinel checks if the timers don’t reach zero, in which case it will terminate the worker and reincarnate a new one.



Scheduler

Twice a minute the scheduler checks for any scheduled tasks that should be starting.

  • Creates a task from the schedule
  • Subtracts 1 from django_q.Schedule.repeats
  • Sets the next run time if there are repeats left or if it has a negative value.



Stop procedure

When a stop signal is received, the sentinel exits the guard loop and instructs the pusher to stop pushing. Once this is confirmed, the sentinel pushes poison pills onto the task queue and will wait for all the workers to exit. This ensures that the task queue is emptied before the workers exit. Afterwards the sentinel waits for the monitor to empty the result queue and the stop procedure is complete.

  • Send stop event to pusher
  • Wait for pusher to exit
  • Put poison pills in the Task Queue
  • Wait for all the workers to clear the queue and stop
  • Put a poison pill on the Result Queue
  • Wait for monitor to process remaining results and exit
  • Signal that we have stopped



References:

https://django-q.readthedocs.io/en/latest/

Generating Diagrams block, aws, redis etc

the link in the references is a nice tool to generate some of the diagrams. 

references:

https://diagrams.mingrammer.com/docs/getting-started/examples

What is Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic


Redis simplifies your code by enabling you to write fewer lines of code to store, access, and use data in your applications. For example, if your application has data stored in a hashmap, and you want to store that data in a data store – you can simply use the Redis hash data structure to store the data.


Essentially, Redis is a NoSQL in-memory data structure store that can persist on disk. It can function as a database, a cache, and a message broker. Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence.


Redis is a data structure server. As a key-value data store, Redis is similar to Memcached, although it has two major advantages over that option: support of additional datatypes and persistence. ... All of the data is stored in RAM, so the speed of this system is phenomenal, often performing even better than Memcached.


It's fully managed. Google fully manages administrative tasks for Redis instances such as hardware provisioning, setup and configuration management, software patching, failover, monitoring and other nuances that require considerable effort for service owners who just want to use Redis as a memory store or a cache


Redis may be slow in scenarios where Not enough memory, generating swapping at the OS level. Too many O(n) operations (like KEYS) executed in the single-threaded engine. Large objects stored in Redis, leading to uncontrolled expansion of the communication buffers. Huge number of simultaneous sessions (>30000)


References 

https://redis.io/

Thursday, June 24, 2021

Communicating between docker containers

In the real world, beyond the realm of the simple hello-world tutorial, running just one container isn’t enough for most apps. A modern application typically consists of a few components – such as a database, a web server, or some microservices.


How to do simple communication between Docker containers, when they are running on the same host (which is sometimes called single-host networking).

Although containers have a level of isolation from the environment around them, they often need to communicate with each other, and the outside world.


Two containers can talk to each other in one of two ways, usually:


Communicating through networking: Containers are designed to be isolated. But they can send and receive requests to other applications, using networking.

For example: a web server container might expose a port, so that it can receive requests on port 80. Or an application container might make a connection to a database container.


Sharing files on disk: Some applications communicate by reading and writing files. These kinds of applications can communicate by writing their files into a volume, which can also be shared with other containers.



Communication between containers with networking

For example, an application might call a REST or GraphQL API, or open a connection to a database.

Containers are ideal for applications or processes which expose some sort of network service. The most well-known examples of these kinds of applications are:

  • Web servers - e.g. Nginx, Apache
  • Backend applications and APIs - e.g. Node, Python, JBoss, Wildfly, Spring Boot
  • Databases and data stores - e.g. MongoDB, PostgreSQL


If you are running more than one container, you can let your containers communicate with each other by attaching them to the same network.

Docker creates virtual networks which let your containers talk to each other. In a network, a container has an IP address, and optionally a hostname.

You can create different types of networks depending on what you would like to do. 


  • The default bridge network, which allows simple container-to-container communication by IP address, and is created by default.
  • user-defined bridge network, which you create yourself, and allows your containers to communicate with each other, by using their container name as a hostname.


The simplest network in Docker is the bridge network. It’s also Docker’s default networking driver.

A bridge network gives you simple communication between containers on the same host.


When Docker starts up, it will create a default network called… bridge. 🤔 It should start automatically, without any configuration required by you.

From that point onwards, all containers are added into to the bridge network, unless you say otherwise.


In a bridge network, each container is assigned its own IP address. So containers can communicate with each other by IP.


How to use the default bridge network


Check that the bridge network is running: You can check it’s running by typing docker network ls. This should show the bridge network in the list.


docker network ls

NETWORK ID     NAME      DRIVER    SCOPE

acce5c7fd02b   bridge    bridge    local

a6998b3cf420   host      host      local

d7f563b21fc6   none      null      local



Start your containers: Start your containers as normal, with docker run. When you start each container, Docker will add it to the bridge network.

(If you prefer, you can be explicit about the network connection by adding --net=bridge to the docker run command.)



Address another container by its IP address: Now one container can talk to another, by using its IP address.



Below are some useful commands on this


docker inspect <container_id> | grep IPV4Address

            "IPAddress": "172.17.0.2",



Here is an example pf nginx container. Then I’ll start a busybox container alongside nginx, and try to make a request to Nginx with wget:


# Start an nginx container, give it the name 'mynginx' and run in the background

$ docker run --rm --name mynginx --detach nginx


# Get the IP address of the container

$ docker inspect mynginx | grep IPAddress

            "IPAddress": "172.17.0.2",


# Or, if you have 'jq' installed - here's a funky way to get the IP address

$ sudo docker inspect mynginx | jq '.[].NetworkSettings.Networks.bridge.IPAddress'

"172.17.0.2"


# Run busybox (a utility container). It will join the bridge network

$ docker run -it busybox sh


# Fetch the nginx homepage by using the container's IP address

busybox$ wget -q -O - 172.17.0.2:80

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

<style>



To see if the container is in the bridge network, the command is below 


sudo docker inspect bridge: 


This will have section like this below in the response. However, the application that was started from the docker-compose, running this command showed nothing in the container dictionary. May be that the container networking in docker-compose is different. 


 "Containers": {

        "1cdc34001e9f5b109836d...": {

            "Name": "vibrant_tesla",   # This is my busybox container

            "EndpointID": "6d51e27f9277bf2...",

            "MacAddress": "02:42:ac:11:00:03",

            "IPv4Address": "172.17.0.3/16",

            "IPv6Address": ""

        },

        "dbb6b814d0f11bfcad11e...": {

            "Name": "mynginx",         # This is my nginx container

            "EndpointID": "aa65052c8c4e26fd...",

            "MacAddress": "02:42:ac:11:00:02",

            "IPv4Address": "172.17.0.2/16",

            "IPv6Address": ""

        }

    },



One point to note is, in the default bridge scenario, each container can see each other. So a more sensible option will be user defined network. 


References:

https://www.tutorialworks.com/container-networking/