Forgather Syntax Reference¶
Forgather defines a domain-specific language for the dynamic construciton of Python objects using a combination of Jinja2, YAML, and a few extensions.
This guide will focus on the extensions to these languages. For details on YAML and Jinja2, see:
Targets¶
At the root-level of a Forgather configuration is a mapping, where each name defines a constructable "target" object.
In the following example, two targets are defined, "a_string" and "a_number."
Anchors and Aliases¶
We make heavy use of YAML anchors and aliases, which allow us to define a constructable object which can be used elsewhere.
training_config: &train_config
batch_size: 32
max_steps: 100
model_config: &model_config
hidden_dimension: 128
layers: 12
config:
train: *train_config
model: *model_config
In the above example, we define the "anchors" "train_config" and "model_config," which are also constructable targets; the names do not have to match. These anchors are then used as "aliases" in the definition of the "config" target.
Tags¶
Tags identify the native data type of the node and being with '!' YAML has a number of built-in tages, for example, if one wanted to specify that something is a floating-point number, rather than an int:
# This specified that "scale" is a float, with a value of 2.0
# Without this specificaiton, the data-type would default to 'int'
scale: !!float 2
Forgather makes extensive use of custom tags to define Python object types. For example:
# This defines a 2x2 random random (normal distribution) PyTorch Tensor
random_matrix: !call:torch:randn [2, 2]
For details, see below.
Jinja2¶
The Preprocessor¶
There is a custom Jinja2 preprocessor which implemnts an extended version of Jinja2's Line Statements. These are implemented via regex substition, where the match is converted to normal Jinja syntax.
- ## : Line Comment
- -- : Line Statement
- << : Line Statement w/ left-trim
- >> : Line Statement w/ right-trim
- == : Print Command
- => : Print Command w/ right-trim
Example Input:
## If 'do_loop' is True, then output a list of numbers.
-- if do_loop:
-- for i in range(how_many): ## Loop 'how_many' times.
== '- ' + i|string
-- endfor
<< endif
Is translated to:
{# If 'do_loop' is True, then output a list of numbers. #}
{% if do_loop: %}
{% for i in range(how_many): %}
{{ '- ' + i|string }}
{% endfor %}
{%- endif %}
Output, when passed: do_loop=True, how_many=3
Normal Jinja2 syntax works just fine too. I just find that the normal syntax is visually difficult to parse (without syntax-highlighting) and is awkward to type.
More Formally
line_comment = r'(.*)\s+#{2,}.*'
line_statement = r'\s*(--|<<|>>|==|=>)\s(.*)'
Substitutions:
{
'--': r"{% " + re_match[2] + r" %}
'<<': r"{%- " + re_match[2] + r" %}"
'>>': r"{% " + re_match[2] + r" -%}"
'==': r"{{ " + re_match[2] + r" }}"
'=>': r"{{ " + re_match[2] + r"|trim('\n')}}"
}
Toml Style Blocks¶
You can use toml style syntax for defining Jinja blocks.
If you are not familiar with what Jinja2 blocks are, they define document sections which can be overriden in derived templates.
becomes...
{% block block_name %}
content
{% block nested_block %}
nested block content
{% endblock nested_block %}
{% endblock block_name %}
{% block another_block %}
more content
{% endblock another_block %}
The regular expression for these blocks is:
Formatting options follow the block name. Right now, the only optin is '!', which will trim all leading and training whitespace from the block.
# "filter "trim"
[block_name!]
content
# becomes...
{% filter trim %}{% block block_name %}
content
{% block block_name %}{% endfilter %}
Jinja2 Globals¶
A number of globals have been introduced to the Jinja2 environment to assist with pre-processing.
- isotime() : Returns ISO formatted local-time, with 1-second resolution ("%Y-%m-%dT%H:%M:%S")
- utcisotime() : As with isotime(), but UTC time.
- filetime(): Generates a local-time string suitable to be concatenated with a file-name. ("%Y-%m-%dT%H-%M-%S")
- utcfiletime() : As filetime(), but in UTC time.
- now() : Get datetime.datetime.now()
- utcnow() : Get datetime.datetime.utcnow()
- joinpath(*names) : Join a list of file-path segments via os.path.join()
- normpath(path) : Normalize a file path; os.path.normpath()
- abspath(path) : Convert path to absolute path; os.path.abspath()
- relpath(path) : Convert a path to a relative path; os.path.relpath()
- getenv(name, default) : Call os.environ.get(name, default)
- repr(obj) : Get Python representation of object; repr()
- modname_from_path(module_name) : Given a module file path, return the module name
- user_home_dir() : Return absolute path of user's home directory
- getcwd() : Get the current working directory
- forgather_config_dir() : Get the platform-specific config directory for Forgather.
The following functions from https://pypi.org/project/platformdirs/ - user_data_dir() - user_cache_dir() - user_config_dir() - site_data_dir() - site_config_dir()
Jinja2 Filters¶
toyaml
The 'toyaml' filter converts Jinja2 variables into YAML compatible syntax, with an optional default value. If no default is specified and the variable is undefined, it will raise an error.
As an example, consider the following definition:
If 'rope_scaling' is defined in Python as:
rope_scaling = {
"factor": 32.0,
"high_freq_factor": 4.0,
"low_freq_factor": 1.0,
"original_max_position_embeddings": 8192,
"rope_type": "llama3",
}
rendered_template = template.render(rope_scaling=rope_scaling)
The rendered output will be:
rope_scaling: {factor: 32.0, high_freq_factor: 4.0, low_freq_factor: 1.0, original_max_position_embeddings: 8192, rope_type: llama3}
The same template, if rope_scaling is Undefined, will render as:
Note that this correctly translated None to "null"
Custom File Loader¶
A custom loader, derived from the FileSystemLoader, is defined. This loader has a syntax for splitting a single loaded template file into multiple sub-templates.
The primary use-case for this syntax is template inheritance, which disallows multiple-inheritance. If you inherit from a template and include a template which is derived from another, Jinja2 does not allow you to direclty override blocks from the included template. You can get around this by creating another template, which overrides the desired blocks, and including it the top-level template.
Normally, this would require creating another template file, but who needs that!? It's easier to maintain just one config file.
## This is the main template
-- extends 'base_template.jinja'
## Override block 'foo' from 'base_template.jinja'
-- block foo
-- include 'foo.bar' ## Include the sub-template
-- endblock
#--------------------- foo.bar ---------------------
## This is a sub-template named 'foo.bar'
-- extends 'some_other_base_template.jinja'
## Override block 'bar' from 'some_other_base_template.jinja'
-- block bar
## ... stuff
-- endblock
More formally, the syntax for splitting a document is:
This just says that it starts with "#", followed by at least 3 '-', a space, the name of the template, a space and at least 3 more '-'. Minimally, written as:
Note: You can't split a template defined via a Python string, as this bypasses the Loader; only file templates may be split like this.
YAML¶
Dot-Name Elision¶
Sometimes you may wish to define an dependency, without making it a constructable target on its own. Any target starting with a dot is automatically removed from the set of valid targets.
In the following example, only "main" is a valid construction target, while the anchors may still be referenced elsewhere.
# Define points
.define: &pt1 { x: 0, y: 0 }
.define: &pt2 { x: 5, y: 0 }
.define: &pt3 { x: 0, y: 5 }
main:
# A list of lines, each defined by a pair of points.
- [ *pt1, *pt2 ]
- [ *pt2, *pt3 ]
- [ *pt3, *pt1 ]
Constructed graph...
graph()
{'main': [[{'x': 0, 'y': 0}, {'x': 5, 'y': 0}],
[{'x': 5, 'y': 0}, {'x': 0, 'y': 5}],
[{'x': 0, 'y': 5}, {'x': 0, 'y': 0}]]}
While not apparent from the representation, the points in the lines are not copies, they are all references to the original three points from the definition. There are only three point objects present in the graph!
YAML Types¶
Of the standard YAML 1.1 types, only those which can be implicilty (without specifying the tag) are supported
YAML 1.1 Tag : Python Type / Examples - !!null : None - null - !!bool : bool - True - False - !!int : int - 2 - -6 - !!float : float - 2.0 - 1.2e-4 - !!str : str - "Hello" - world - !!seq : list - [ 1, 2, 3 ] - !!map : dict - { x: 1, y: 12 }
The following standard types are presently unsupported: - !!binary - !!timestamp - !!omap, !!pairs - !!set -- TODO: Implement me!
Complex types are instead supported through Forgather specific tags:
!tuple : Named Tuple¶
Syntax: !tuple[:@name] \<sequence>
Construct a named Python tuple from a YAML sequence
!list : Named List¶
Syntax: !list[:@name] \<sequence>
Construct a named Python list from a YAML sequence
!dict : Named Dictionary¶
Syntax: !dict[:@\<name>] \<mapping>
Construct a named Python dict from a YAML mapping
!dlist : Named Dictionary-List¶
Syntax: !dlist[:@\<name>] \<mapping>
Construct a Dictionary-List. What is that, you may ask? It's a dictionary which resolves to a list type.
Why would you need such a thing? YAML allows us to override an existing value in a dictionary by redeclaring it, which is something we make extensive use of in our templates. List can only be extended, but not overriden. By declaring a list as a dictionary, we can override previous list items by redeclaring them by name -- or erase them from the list by setting their value to Null (~)
This also allows one to declare an empty list, which can be extended later, like this
!var¶
Syntax: !var "\<var-name>" | { name: \<var-name>, default: \<default-value> }
This declares a global variable, which can be substituted anywhere in the graph.
document = """
point: !dict
x: !var "x" # Define a variable named 'x'
y: !var # Define a variable named 'y' with a default value of 16
name: y
default: 16
"""
The global context is passed in as the special 'context_vars' argument, a dictionary, when constructng the graph.
!call¶
Alias: !singleton
Synatx: !call:\<import-spec>[@\<name>] (\<sequence> | \<mapping> | ({ args: \<sequence>, kwargs: \<mapping> }))
This is a callable object with only a single instance; any aliases refers to the same object instance.
# Construct three random ints, all having the same value.
- &random_int !call:random:randrange:@random_int [ 1000 ]
- *random_int
- *random_int
The "SingletonNode" will generally be your 'go-to' for constructing objects, as the symantics mirror what is expected for YAML anchors and aliases.
However, there are a few exceptions...
!factory¶
Synatx: !factory:\<import-spec>[@\<name>] (\<sequence> | \<mapping> | ({ args: \<sequence>, kwargs: \<mapping> }))
This is a callable object which instantiates a new instance everywhere it appears in the graph.
# Construct three random ints, all (probably) having different values.
- &random_int !factory:random:randrange [ 1000 ]
- *random_int
- *random_int
Constructed...
!partial¶
Alias (deprecated): !lambda
Synatx: !partial:\<import-spec>[@\<name>] (\<sequence> | \<mapping> | ({ args: \<sequence>, kwargs: \<mapping> }))
This constructs a callable object with the same symantics of a Python partial function, where the provided positional and keyword arguments are passed to the function. If additional argmuents are given, the positional-args are appended and the keyword-args are merged.
See: https://docs.python.org/3/library/functools.html
!meta¶
Syntax: !meta:\<import-spec>[@\<name>] (\<sequence> | \<mapping> | ({ args: \<sequence>, kwargs: \<mapping> }))
A MetaNode is a specialized SingletonNode that interrupts the normal depth-first
construction of the configuration graph. When a MetaNode is encountered, its child
nodes are not constructed first. Instead, the raw (unconstructed) graph below the
!meta tag is passed directly to the callable for processing.
This allows the callable to inspect or transform the configuration graph before
construction continues. The primary use case is model code generation, where the
configuration below the !meta tag describes a model architecture and the callable
converts that description into dynamically generated Python code used for model
construction.
In principle, !meta can be used for any graph transformation -- the callable receives
the raw configuration, can modify it arbitrarily, and returns the result that replaces
the !meta node in the graph.
# Model code generation: the config below !meta describes the model architecture.
# The callable generates Python source code from this description.
model: !meta:forgather.codegen:code_gen
[model_definition]
type: "transformer"
hidden_size: 512
num_layers: 6
CallableNodes¶
SingletonNode, FactoryNode, and FactoryNode are all instances of the abstract-base-class "CallableNode." A CallableNode can call any Python function, including class constructors. As Python differentiates between positional and keyword args, we provide two options for differentiating between the two:
Explicit:
Implicit, by name:
Positional args are matches via regex and automatically sorted, thus they can be defined in any order, allowing one to override or extend the list of positional arguments by appending to the mapping.
The syntax is pretty flexible...
- !call:torch:tensor
- 2
- 2
- !call:random.binomialvariate { n: 1, p: 0.5 }
- !call:torch.randn [2, 2]
If the Callable does not require arguments, you can ommit them.
CallableNode Tag Syntax¶
The part of the YAML tag after the first ':' provides the information required to locate and import the requested Callable.
In the simplest case, a built-in Python callable just needs to specify the name of the built-in.
When the Callable is defined in a module, a second ':' is used to seperate the module name from the name within the module.
You can also dynamically import a name from a file.
# See: https://docs.python.org/3/library/operator.html
!singleton:/path/to/my/pymodule.py:MyClass [ "foo", "bar" ]
When using a file-import, which itself has relative imports, you will need to specify which directories to search for relative imports:
# See: https://docs.python.org/3/library/operator.html
!singleton:/path/to/my/pymodule.py:MyClass
args: [ "foo", "bar" ]
kwargs:
submodule_searchpath:
- "/path/to/my/"
- "/path/to/shared/modules/"
Named Callable Nodes¶
CallableNodes may be given an explcit name. The name servers the same purpose as the YAML anchor/alias, but PyYaml does not make this information available through the tag API. While feasible to hack PyYaml, doing so is risky. For now, there is a somewhat redundant interface for specitying node names.
When a node has been assigned an explicit name, it will always be rendered as an explciit definition in the Python and Yaml code generators, as to improve readability. Doing so is entirely optional.
A callable node's tag may end with '@\<name>' which will assign a name to the node.
.define: &foobar !singleton:dict@foobar
foo: 1
bar: 2
baz: |
She sells sea shells
by the sea shore
main:
- *foobar
When rendered as Python:
def construct(
):
foobar = {
'foo': 1,
'bar': 2,
'baz': (
'She sells sea shells\n'
'by the sea shore\n'
),
}
return {
'main': [
foobar,
],
}
And without the name, the object definition becomes anonymous: