misc/blackmesa/records

Django Records

Create arbitrary classes instead of django models for usage in layered architectures.

Index

• Records fetching Data into arbitrary classes
• Deeper Example
• TODO: setting up your models and querysets

Records fetching Data into arbitrary classes

The records() QuerySet and Manager command

The idea behind records() is to instead of directly transforming a Model into a dataclass, to fetch data from the database with a values() like call. However instead dictionaries that values() creates, it produces arbitrary classes or dataclasses, so, "records". It completely skips instantiation of a django model instance, and comes with tools to make it easy to handle initialization data, so that you can automate even production of immutable or nested data structures.

Out of the box, records assumes, you want to use dataclasses.dataclass

Usage:

    SomeModel.objects.filter(...).records(DataClass, 'field1', 'field2', annotation=F(...), adjunct=Adjunct(...)) 

How Records works and what it returns

Just like Values() returns an iterator, that will work with database calls in chunks, Records() returns an iterator, that will work exactly the same mechanically, but instead of producing dictionaries, it produces whatever you define as a "record".

This means, that you can also use things like .first() chained (which will return a single record)

records() works very similar to Django .values():

• args are fields that are selected.
• kwargs that are expressions are annotated into the key as field

However it also differs:

• The first (and only first) argument in args can be a target class to produce. See: RecordHandler.
• you should not define the fields, which are part of your target class for most handlers. They are added automatically to the values() call, by the handler.
• kwargs can contain a special kind of class, called an Adjunct, which is not included in the SQL, and allows "local data" to be inserted into the resulting class produced. See: Adjunct.

RecordHandler

The Dataclasses produced are handled by the RecordHandler class. This allows to adjust the creation of the dataclass, to whatever library you want to use. At the moment, an example for handling a dictionary, and to a dataclasses.dataclass.

Todo: Pydantic, attrs, namedtuple.

By Default, it expects dataclasses.dataclass, and uses the RecordDataclassHandler.

The whole class is kept simple, so that you can write your own RecordHandler if you work with a different kind of dataclass.

Instead of providing a record class, you can also provide an instantiated record handler. For example, you could use RecordDictHandler(), and you would get dictionaries from records(). Which essentially would be the same output as values(). Except that you can use Adjuncts.

Adjunct

Just like Django Expressions can be used to annotate keys in the model that are retrieved, so can Adjuncts be used to circumvent this mechanic, and insert local data into your target class. You might want to use this, if e.g. the dataclass you create is immutable, or the dataclass you use has required fields, that need data when you create the class, but the data is not part of your database query.

Various Adjunct mechanics are available:

• Adjunct itself simply carries some data and inserts it into every model. e.g. .records(data=Adjunct(1)) will set the field data always to 1.
• Lambda allows to use a callable as argument, which gets called when setting the field on the model. e.g. .records(data=Lambda(lambda entry: 'x' in entry))
• Skip allows you to skip a field. This is needed, as records() would include all fields on a dataclass, without knowing if it is optional.
• Callback allows you to call a function as a callback at creation - if the callback returns anything else than None, it is used as initializer for the production of the object.

Just as with expressions, you can of course also write your own Adjunct classes, by subclassing BaseAdjunct.

default_record defininition

If records() is not provided a class or handler, you can define a default either on the Manager or QuerySet, by using _default_record. You can also put this on the model class. If a record class is not found, an Exception is raised.

Deeper Example

Assume we have a model like so in models.py (made simpler to read by leaving out details):

class MyModel:
    name = models.CharField()
    age = models.IntegerField()
    parent = models.ForeignKey('self')
    data = models.TextField()
    
    objects = records.RecordManager()

Now let's assume we have a dataclass in domain.py:

@dataclass
class MyRecord:
    name: str
    age: int
    parent_id: int
    parent_name: str
    has_data: bool

Building this record, which for now we assume will be a frozen dataclass in the future, would be done like this:

queryset = MyModel.objects.filter(...)\
               .records(MyRecord, 'data', age=Adjunct(0), parent_name=F('parent__name'), has_data=Lambda(lambda d: d.get('data') is not None))

list(queryset) will return data like this: [MyRecord(name=..., age=0, parent_id=..., parent_name=..., has_data=...), ...]

What has been done here?

• the field data was not on MyRecord, but needed by has_data, so it was included in the values() call by providing it as a string argument.
• name will be automatically inserted. the parent_id will be picked up as well, containing the primary key of the parent. They do not need to be defined in the records() call.
• age however will always be 0, and not inserted into the db call.
• the parent_name was, just like in values(), a Django Expression as kwarg. it will be written into the appropriate field.
• has_data finally will get it's data by calling the lambda function for each fetch, when the record is instantiated. for that it will extract whether the data entry returned some value from the db, which we included earlier for this reason. Of course this would have been possible with a db call as well, but this is to show the mechanic.

similarly, you could just call queryset.first() and retrieve the first MyRecord entry, or None.