Python building C and C++ extensions

A C extension for CPython is a shared library (e.g. a .so file on Linux, .pyd on Windows), which exports an initialization function.

To be importable, the shared library must be available on PYTHONPATH, and must be named after the module name, with an appropriate extension. When using distutils, the correct filename is generated automatically.

The initialization function has the signature:

PyObject* PyInit_modulename(void)

It returns either a fully-initialized module, or a PyModuleDef instance. See Initializing C modules for details.

For modules with ASCII-only names, the function must be named PyInit_<modulename>, with <modulename> replaced by the name of the module. When using Multi-phase initialization, non-ASCII module names are allowed. In this case, the initialization function name is PyInitU_<modulename>, with <modulename> encoded using Python’s punycode encoding with hyphens replaced by underscores. In Python:

def initfunc_name(name):

    try:

        suffix = b'_' + name.encode('ascii')

    except UnicodeEncodeError:

        suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')

    return b'PyInit' + suffix

It is possible to export multiple modules from a single shared library by defining multiple initialization functions. However, importing them requires using symbolic links or a custom importer, because by default only the function corresponding to the filename is found. See the “Multiple modules in one library” section in PEP 489 for details.

References:

https://docs.python.org/3/extending/building.html

How to create a python whl file ?

Python wheel is an executable pip package. This allows organize all your Python modules in nice and clean way for easy importing the underlying Python function / class in other codes. 

Python module is essentially a script generally consisting of some functions and/or classes, which can be referenced in other codes to make them concise, more readable, and easy to upgrade/enhance/maintain the modules — as all of it is kept in a single place.

Once a Python Wheel is created, you can install it (file format with .whl extension) using simple pip install [name of wheel file].

Below is steps to create simple and basic wheel file

Keep all the modules (python scripts), packages (folders/directories, which contain the modules) in a parent directory. Name the root directory whatever you like, typically something related to a project

Preferably, create an empty .py file named __init__, and place this __init__.py under all the package directories and sub-package / sub-directories. No need to keep this in the root directory. Note that, this is not mandatory, but will be helpful.

Create a file named setup.py and place it in the root directory. Content of this script at the very minimal should include the distribution name, version number, and list of package names. An example below

from setuptools import setup, find_packages

setup(

    # this will be the package name you will see, e.g. the output of 'conda list' in anaconda prompt

    name='ruleswhl',

    # some version number you may wish to add - increment this after every update

    version='1.0',

    # Use one of the below approach to define package and/or module names:

    # if there are only handful of modules placed in root directory, and no packages/directories exist then can use below syntax

    #     packages=[''], #have to import modules directly in code after installing this wheel, like import mod2 (respective file name in this case is mod2.py) - no direct use of distribution name while importing

    # can list down each package names - no need to keep __init__.py under packages / directories

    #     packages=['<list of name of packages>'], #importing is like: from package1 import mod2, or import package1.mod2 as m2

    # this approach automatically finds out all directories (packages) - those must contain a file named __init__.py (can be empty)

    # include/exclude arguments take * as wildcard, . for any sub-package names

    packages=find_packages(),

)

Now from the command line, enter the below to build the whl file 

python setup.py bdist_wheel

The built whl file can be installed as usual using pip 

pip install <whl file>

References

https://medium.com/swlh/beginners-guide-to-create-python-wheel-7d45f8350a94

Django Model How to have a hook to deletion of the model

For the DefaultAdminSite the delete_queryset is called if the user has the correct permissions, the only difference is that the original function calls queryset.delete() which doesn't trigger the model delete method. This is less efficient since is not a bulk operation anymore, but it keeps the filesystem clean.

Below are example Model and ModelAdmins 

models.py

class MyModel(models.Model):

    file = models.FileField(upload_to=<path>)

    def save(self, *args, **kwargs):

        if self.pk is not None:

            old_file = MyModel.objects.get(pk=self.pk).file

            if old_file.path != self.file.path:

                self.file.storage.delete(old_file.path)

        return super(MyModel, self).save(*args, **kwargs)

    def delete(self, *args, **kwargs):

        ret = super(MyModel, self).delete(*args, **kwargs)

        self.file.storage.delete(self.file.path)

        return ret

admin.py

class MyModelAdmin(admin.ModelAdmin):

    def delete_queryset(self, request, queryset):

        for obj in queryset:

            obj.delete()

References:

https://stackoverflow.com/questions/1471909/django-model-delete-not-triggered

Model Meta Options in Django

db_table

Options.db_table

The name of the database table to use for the model:

db_table = 'music_album'

Table names¶

To save you time, Django automatically derives the name of the database table from the name of your model class and the app that contains it. A model’s database table name is constructed by joining the model’s “app label” – the name you used in manage.py startapp – to the model’s class name, with an underscore between them.

For example, if you have an app bookstore (as created by manage.py startapp bookstore), a model defined as class Book will have a database table named bookstore_book.

To override the database table name, use the db_table parameter in class Meta.

verbose_name¶

Options.verbose_name¶

A human-readable name for the object, singular:

verbose_name = "pizza"

If this isn’t given, Django will use a munged version of the class name: CamelCase becomes camel case.

verbose_name_plural¶

Options.verbose_name_plural¶

The plural name for the object:

verbose_name_plural = "stories"

If this isn’t given, Django will use verbose_name + "s".

Read-only Meta attributes¶

label¶

Options.label¶

Representation of the object, returns app_label.object_name, e.g. 'polls.Question'.

label_lower¶

Options.label_lower¶

Representation of the model, returns app_label.model_name, e.g. 'polls.question'.

constraints¶

Options.constraints¶

A list of constraints that you want to define on the model:

from django.db import models

class Customer(models.Model):

    age = models.IntegerField()

    class Meta:

        constraints = [

            models.CheckConstraint(check=models.Q(age__gte=18), name='age_gte_18'),

        ]

Others few are

index_together = [

    ["pub_date", "deadline"],

]

unique_together = [['driver', 'restaurant']]

Indexes to be used in place of index_together. 

class Meta:

        indexes = [

            models.Index(fields=['last_name', 'first_name']),

            models.Index(fields=['first_name'], name='first_name_idx'),

        ]

ordering¶

Options.ordering¶

The default ordering for the object, for use when obtaining lists of objects:

ordering = ['-order_date']

ordering = ['pub_date']

ordering = ['-pub_date']

ordering = ['-pub_date', 'author']

ordering = [F('author').asc(nulls_last=True)]

References:

https://docs.djangoproject.com/en/3.2/ref/models/options/

python Importlib main methods

Import a module. The name argument specifies what module to import in absolute or relative terms (e.g. either pkg.mod or ..mod). If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod).

The import_module() function acts as a simplifying wrapper around importlib.__import__(). This means all semantics of the function are derived from importlib.__import__(). The most important difference between these two functions is that import_module() returns the specified package or module (e.g. pkg.mod), while __import__() returns the top-level package or module (e.g. pkg).

If you are dynamically importing a module that was created since the interpreter began execution (e.g., created a Python source file), you may need to call invalidate_caches() in order for the new module to be noticed by the import system.If you are dynamically importing a module that was created since the interpreter began execution (e.g., created a Python source file), you may need to call invalidate_caches() in order for the new module to be noticed by the import system.

importlib.reload(module)

Reload a previously imported module. The argument must be a module object, so it must have been successfully imported before. This is useful if you have edited the module source file using an external editor and want to try out the new version without leaving the Python interpreter. The return value is the module object (which can be different if re-importing causes a different object to be placed in sys.modules).

When reload() is executed:

• Python module’s code is recompiled and the module-level code re-executed, defining a new set of objects which are bound to names in the module’s dictionary by reusing the loader which originally loaded the module. The init function of extension modules is not called a second time.
  
• As with all other objects in Python the old objects are only reclaimed after their reference counts drop to zero.
  
• The names in the module namespace are updated to point to any new or changed objects.
  
• Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.
  

References:

https://docs.python.org/3/library/importlib.html

Metaprogramming in Python

The term metaprogramming refers to the potential for a program to have knowledge of or manipulate itself. Python supports a form of metaprogramming for classes called metaclasses. Metaclasses are an esoteric OOP concept, lurking behind virtually all Python code. You are using them whether you are aware of it or not.

When the need arises, however, Python provides a capability that not all object-oriented languages support: you can get under the hood and define custom metaclasses

understanding Python metaclasses is worthwhile, because it leads to a better understanding of the internals of Python classes in general. You never know: you may one day find yourself in one of those situations where you just know that a custom metaclass is what you want.

Django Model_meta API 

The model _meta API is at the core of the Django ORM. It enables other parts of the system such as lookups, queries, forms, and the admin to understand the capabilities of each model. The API is accessible through the _meta attribute of each model class, which is an instance of an django.db.models.options.Options object.

Methods that it provides can be used to:

• Retrieve all field instances of a model
• Retrieve a single field instance of a model by name

Field access API

Retrieving a single field instance of a model by name

Options.get_field(field_name)¶

Returns the field instance given a name of a field.

field_name can be the name of a field on the model, a field on an abstract or inherited model, or a field defined on another model that points to the model. In the latter case, the field_name will be (in order of preference) the related_query_name set by the user, the related_name set by the user, or the name automatically generated by Django.

Hidden fields cannot be retrieved by name.

If a field with the given name is not found a FieldDoesNotExist exception will be raised.

Example is given below 

from django.contrib.auth.models import User

User._meta.get_field('username')

A field from another model that has a relation with the current model

User._meta.get_field('logentry')

<ManyToOneRel: admin.logentry>

User._meta.get_field('does_not_exist')

Traceback (most recent call last):

    ...

FieldDoesNotExist: User has no field named 'does_not_exist'

Retrieving all field instances of a model

Options.get_fields(include_parents=True, include_hidden=False)¶

include_parents

True by default. Recursively includes fields defined on parent classes. If set to False, get_fields() will only search for fields declared directly on the current model. Fields from models that directly inherit from abstract models or proxy classes are considered to be local, not on the parent.

include_hidden

False by default. If set to True, get_fields() will include fields that are used to back other field’s functionality. This will also include any fields that have a related_name (such as ManyToManyField, or ForeignKey) that start with a “+”.

from django.contrib.auth.models import User

>>> User._meta.get_fields()

(<ManyToOneRel: admin.logentry>,

 <django.db.models.fields.AutoField: id>,

 <django.db.models.fields.CharField: password>,

 <django.db.models.fields.DateTimeField: last_login>,

 <django.db.models.fields.BooleanField: is_superuser>,

 <django.db.models.fields.CharField: username>,

 <django.db.models.fields.CharField: first_name>,

 <django.db.models.fields.CharField: last_name>,

 <django.db.models.fields.EmailField: email>,

 <django.db.models.fields.BooleanField: is_staff>,

 <django.db.models.fields.BooleanField: is_active>,

 <django.db.models.fields.DateTimeField: date_joined>,

 <django.db.models.fields.related.ManyToManyField: groups>,

 <django.db.models.fields.related.ManyToManyField: user_permissions>)

What is a whl file ?

A WHL file is a package saved in the Wheel format, which is the standard built-package format used for Python distributions. It contains all the files for a Python install and metadata, which includes the version of the wheel implementation and specification used to package it.

The Wheel format was introduced in PEP 427, which is a Python Enhancement Proposal authored by Daniel Holth and accepted in 2012. It was developed as a quicker and more reliable method of installing Python software than re-building from source code every time. WHL files only have to be moved to the correct location on the target system to be installed, whereas a source distribution requires a build step before installation.

References:

https://fileinfo.com/extension/whl#:~:text=A%20WHL%20file%20is%20a,specification%20used%20to%20package%20it.