Project Index¶
import forgather.nb.notebooks as nb
nb.display_project_index(show_available_templates=True)
Project Composition¶
This example demostrates the use of Jinja2 template inheritance.
In addition to YAML, there is a Jinja2 preprocessing stage which allows for things like macros and template inheritance. This can help eliminate unnecessary repition by factoring out the common elements in a set of configurtions.
In this example, we define a base-template ("list_base.yaml") for defining a list and extend the definition for the first configuration, "list.yaml." In the second configuration, "full_list.yaml," we extend the definition of "list.yaml."
We use a list to keep this example simple, but this pattern is used extensively in the main Forgather template library for much more complex use-cases.
Project Setup¶
The project meta-config is much the same as the first example project, although we only specify the default config this time, as the other defaults will work.
Configurations¶
Under "Available Configurations," there are two configs listed:
- list.yaml : A short list, derived from base_list.yaml
- full_list.yaml : Alonger list, derived from list.yaml
Included Templates¶
Note the hierarchical template listing for the selected configuration. You can examine the referenced templates by clicking on the links in the index.
Project Directory: "/home/dinalt/ai_assets/forgather/examples/tutorials/project_composition"¶
Meta Config¶
Meta Config: /home/dinalt/ai_assets/forgather/examples/tutorials/project_composition/meta.yaml
Template Search Paths:
Available Configurations¶
Default Configuration: list.yaml
Available Templates¶
The default configuration, "list.yaml" looks like this:
with open("./templates/configs/list.yaml", "r") as f:
nb.display_codeblock("yaml", f.read())
-- extends "list_base.yaml"
## Override config name
-- block project_meta
== super()
-- set ns.config_name = "A short list"
<< endblock project_meta
-- block list
- Alpha
- Bravo
- Charlie
- Delta
<< endblock list
If you are familiar with Jinja2, the first thing you may notice is that we are using "line-statements," where the following are equivalent:
jinja2
{% extends "list_base.yaml" %}
{# Override config name #}
and this...
-- extends "list_base.yaml"
## Override config name
Both of the above are Jinja2 statements and comments, respectively. You can use either style.
Jinja allows template inheritance, where the "extends" statements indicates that this file is a "child" of "list_base.yaml."
A child template may override the "blocks" of its parent, which replaces the parent's definition. The parent's definition may be included in the child with "super()."
-- block project_meta
== super()
-- set ns.config_name = "A short list"
<< endblock project_meta
Here, we are overriding a block named "project_meta" by appending to the original definition.
Let's take a look at the parent definition.
with open("./templates/list_base.yaml", "r") as f:
nb.display_codeblock("yaml", f.read())
## Create a new namespace
-- set ns = namespace()
## Import formatting marcos
-- from 'formatting.yaml' import h1, sep
## Strip any whitespace generated by the definitions
-- filter trim()
## Define project meta-data
-- block project_meta
-- set ns.config_name = "Anonymous"
-- set ns.config_description = "Construct a list"
<< endblock project_meta
-- endfilter ## filter trim() setup
== h1(ns.config_name)
-- block header
# {{ utcisotime() }}
# Description: {{ ns.config_description }}
# Project Dir: {{ abspath(project_dir) }}
<< endblock header
== '\n' + sep()
meta:
-- block meta
name: "{{ ns.config_name }}"
description: "{{ ns.config_description }}"
<< endblock meta
main:
-- block list required
## Define a list here
<< endblock list
Let's break some of this down...
The first statement defines a Jinja2 namespace:
-- set ns = namespace()
What is the purpose of the namespace?
Please keep in mind that it is not possible to set variables inside a block and have them show up outside of it.
A namespace allows us to side-step this restriction, thus if a block assigns a variable in a namepace, the change will be visisble outside of that block.
-- from 'formatting.yaml' import h1, sep
In the above, we are importing macros from a template named 'formatting.yaml'
jinja2
{%- macro h2(name='Heading 2') %}{{ "{:#^40}".format(' ' + name + ' ') }}{% endmacro %}
{%- macro h3(name='Heading 3') %}{{ '# **' + name + '**' }}{% endmacro %}
{%- macro h4(name='Heading 4') %}{{ '# ' + name }}{% endmacro %}
{%- macro sep() %}{{ '#' + "{:-^39}".format('') }}{% endmacro %}
{%- macro h1(name='Title') %}
{{ sep() }}
{{ '# ' + "{:^39}".format(' ' + name + ' ') }}
{{ sep() }}
{%- endmacro %}
Specifically, we are importing the macros named 'h1' and 'sep,' short for 'Heading-1' and 'separator,' which we will use for text formatting.
-- filter trim()
...
-- endfilter
This pair of matched statements filters extra whitespace from the resulting output.
-- block project_meta
-- set ns.config_name = "Anonymous"
-- set ns.config_description = "Construct a list"
<< endblock project_meta
This defines a block of text, with the tag 'project_meta.' This block sets a couple of Jinja2 variables, which can be overriden by redefining them in a child template.
Note the '<<' line-statement. This functions as a normal line-statement, except it also strips empty lines on the side where the "arrows" are pointing. This is not required, but can make the resulting output a bit cleaner.
== h1(ns.config_name)
-- block header
# {{ utcisotime() }}
# Description: {{ ns.config_description }}
# Project Dir: {{ abspath(project_dir) }}
<< endblock header
The first line uses the 'h1' macro. The '==' means that this replaces that line with the output of the statement. It is equivalent to:
jinja2
{{ h1(ns.config_name) }}
In this case, the macro substitution will result in this output:
#---------------------------------------
# A short list
#---------------------------------------
The remaiing lines in the block generate yaml comments, with the contents substitued by Jinja2 variables and functions.
These following two lines are pure YAML. They define dictionary keys at the root of the configuration and correspond to the available output-targets of the configuration.
meta:
...
main:
A Small Digression¶
In some of the examples, you may encouter encounter dictionary keys prefixed with a dot.
.define: &something "A string"
The dot specifies that the key will be hidden from the list of output-targets. Even though it is no longer a target, it still has a purpose; it has a Yaml anchor, "&something," which can be substituted elsewhere in the configuraiton. This is used to define something which may be used more than once in the output, but cannot be directly constructed e.g.
my_list:
- *something
my_dict:
something: *something
The above defines two output targets, my_list and my_dict, which both include the same instance of "something," whatever than happens to be.
The Project Class¶
The high-level interface for constructing the objects defined by a project configuration is the 'Project' class. The project object has the following dataclass members:
- config_name : The name of the selected configuration; automatically populated with the default, if unspecified.
- project_dir : The absolute path to the project directory.
- meta : The project's meta-config.
- environment : The projects config envrionment.
- config : The constructed node-graph, representing the configuration.
- pp_config : The pre-processed configuration.
from forgather import Project
from pprint import pp
# This load the default configuration into the project object, but an actual instance has not yet been constructed.
proj = Project()
pp(proj)
Project(config_name='list.yaml',
project_dir='/home/dinalt/ai_assets/forgather/examples/tutorials/project_composition',
meta=MetaConfig(project_dir='/home/dinalt/ai_assets/forgather/examples/tutorials/project_composition',
name='meta.yaml',
meta_path='/home/dinalt/ai_assets/forgather/examples/tutorials/project_composition/meta.yaml',
searchpath=['/home/dinalt/ai_assets/forgather/examples/tutorials/project_composition/templates'],
system_path=None,
config_prefix='configs',
default_cfg='list.yaml',
config_dict={'default_config': 'list.yaml'},
workspace_root='/home/dinalt/ai_assets/forgather'),
environment=<forgather.config.ConfigEnvironment object at 0x7f16823365c0>,
config={'meta': {'name': 'A short list',
'description': 'Construct a list'},
'main': ['Alpha', 'Bravo', 'Charlie', 'Delta']},
pp_config='\n'
'\n'
'#---------------------------------------\n'
'# A short list \n'
'#---------------------------------------\n'
'# 2025-06-19T23:10:36\n'
'# Description: Construct a list\n'
'# Project Dir: '
'/home/dinalt/ai_assets/forgather/examples/tutorials/project_composition\n'
'#---------------------------------------\n'
'\n'
'\n'
'meta:\n'
' name: "A short list"\n'
' description: "Construct a list"\n'
'\n'
'main:\n'
' - Alpha\n'
' - Bravo\n'
' - Charlie\n'
' - Delta')
Display Project Attributes¶
There are a number of helper functions in the Noteboot module which can help with rendering project attributes.
import forgather.nb.notebooks as nb
nb.display_meta(proj.meta)
nb.display_codeblock("yaml", proj.pp_config)
#---------------------------------------
# A short list
#---------------------------------------
# 2025-06-19T23:10:36
# Description: Construct a list
# Project Dir: /home/dinalt/ai_assets/forgather/examples/tutorials/project_composition
#---------------------------------------
meta:
name: "A short list"
description: "Construct a list"
main:
- Alpha
- Bravo
- Charlie
- Delta
Display the Node Graph¶
The node-graph (proj.config) defines how to construct the defined object.
A simple config, like the one defined in this project, is easy enough to interpret by just printing it. It may make it a little easier, if we add Python syntax highlighting.
nb.display_codeblock("python", proj.config)
{'meta': {'name': 'A short list', 'description': 'Construct a list'}, 'main': ['Alpha', 'Bravo', 'Charlie', 'Delta']}
Display as YAML¶
The node-graph can be rendered as YAML, which may be helpful for more complex graphs.
from forgather.yaml_encoder import to_yaml
nb.display_codeblock("yaml", to_yaml(proj.config))
meta:
name: 'A short list'
description: 'Construct a list'
main:
- 'Alpha'
- 'Bravo'
- 'Charlie'
- 'Delta'
Display as Python Code¶
Another option is to render the code graph as the equivalent Python code.
from forgather.codegen import generate_code
nb.display_codeblock("python", generate_code(proj.config))
def construct(
):
return {
'meta': {
'name': 'A short list',
'description': 'Construct a list',
},
'main': [
'Alpha',
'Bravo',
'Charlie',
'Delta',
],
}
Object Construction¶
Calling the project object, without arguments, will instantiate the 'main' target object.
phonetic_alphabet = proj()
pp(phonetic_alphabet)
['Alpha', 'Bravo', 'Charlie', 'Delta']
Calling the project object with a single positional string argument will construct and return the specified target.
proj("meta")
{'name': 'A short list', 'description': 'Construct a list'}
Calling the project object with an iterable of strings will return a dictionary of the specified targets.
proj(["main", "meta"])
{'meta': {'name': 'A short list', 'description': 'Construct a list'},
'main': ['Alpha', 'Bravo', 'Charlie', 'Delta']}
If a target does not exist, the corresponding key will be absent from the output.
proj(["main", "foo"])
{'main': ['Alpha', 'Bravo', 'Charlie', 'Delta']}
Calling the project with individual string arguments returns an iterable.
main, meta = proj("main", "meta")
main, meta
(['Alpha', 'Bravo', 'Charlie', 'Delta'],
{'name': 'A short list', 'description': 'Construct a list'})
Selecting a Project Configuration¶
If there is more than one available configuration, the configuration can be passed as an argument.
proj = Project("list.yaml")
proj()
['Alpha', 'Bravo', 'Charlie', 'Delta']
Alternatively, the configuraiton for ab existing project can be changed.
proj.load_config("full_list.yaml")
proj()
['Alpha', 'Bravo', 'Charlie', 'Delta', 'Echo', 'Foxtrot', 'Golf', 'Hotel', 'India', 'Julliet', 'Kilo', 'Lima', 'Mike', 'November', 'Oscar', 'Papa', 'Quebec', 'Romeo', 'Sierra', 'Tango', 'Uniform', 'Victor', 'Whisky', 'X-Ray', 'Yankee', 'Zulu']
Code Execution¶
How can I execute dynamically generated code?
Please note that this is not how this works internally when constructing a configuration; the node graph is directly converted into the constructed object, without being first translated into code. There are still a few corner-cases where the generated code does not do exactly the same thing a directly constructing the configuration.
The main known issue is that arguments passing arguments to lambdas does not work in generated code, but works when directly constructed.
from forgather.codegen import generate_code
# The 'config' attribute of the project is the raw configuration node-graph.
# The graph can be converted to executable Python code with 'generate_code'
# Note that we can independenlty generate code for any node in the graph.
generated_code = generate_code(proj("main"))
nb.display_codeblock("python", generated_code, "## Generated Code\n")
# Calling 'exec' on the code is roughly equivlant to pasting the code into a cell
# and executing the cell. With this configuration, it outputs a function named
# 'construct,' which can be called to construct the configuration.
exec(generated_code)
phonetic_alphabet = construct()
nb.display_codeblock("python", phonetic_alphabet, "## Code Output\n")
Generated Code¶
def construct(
):
return [
'Alpha',
'Bravo',
'Charlie',
'Delta',
'Echo',
'Foxtrot',
'Golf',
'Hotel',
'India',
'Julliet',
'Kilo',
'Lima',
'Mike',
'November',
'Oscar',
'Papa',
'Quebec',
'Romeo',
'Sierra',
'Tango',
'Uniform',
'Victor',
'Whisky',
'X-Ray',
'Yankee',
'Zulu',
]
Code Output¶
['Alpha', 'Bravo', 'Charlie', 'Delta', 'Echo', 'Foxtrot', 'Golf', 'Hotel', 'India', 'Julliet', 'Kilo', 'Lima', 'Mike', 'November', 'Oscar', 'Papa', 'Quebec', 'Romeo', 'Sierra', 'Tango', 'Uniform', 'Victor', 'Whisky', 'X-Ray', 'Yankee', 'Zulu']