Forgather Projects Overview¶
The high-level Forgather API organizes a collection of configurations (experiments) into a common "Project," where a Forgather Project is defined by a directory with a "meta.ymal" file in it. At a minimum, the meta-file is expected to have a list of directories to search for templates.
In this project's meta file, only a single directory is named, the "templates" directory. If more than one directory is listed, they will be searched in the order they occur in this list.
searchpath: [ "joinpath(project_dir, 'templates')" ]
If you look inside of the directory, you will find "project.yaml," which defines global defaults for our configurations. This convention is not required, but it is used in most of the example projects.
You will also see a "configs" directory. This corresponds to this line in the meta-config file:
config_prefix: "configs"
The directory named by "config_prefix" is special, in that it is the only location where the project will search for configuration files.
Finally, the meta specifies:
default_config: "default.yaml"
This names the default configuration file in the directory named by the 'config_prefix.' Thus, the default configuration is located at "./templates/configs/default.yaml"
To recap, a minimal Project contains a "meta.yaml" file, which specifies where to search for templates, and one or more configuration files found in a directory matching the 'config_prefix.'
Workspace¶
While not used in this example, it is worth mentioning that the meta.yaml file can implicitly include or extend templates found in the nearest "forgather_workspace" direcory. This is found by recursively searching parent directories. To see an example of how this is used, see "tiny_experiments," in the "examples" directory.
Project Index¶
With the help of the 'meta.yaml' file, we can generate an index for the Project like this:
import forgather.nb.notebooks as nb
nb.display_project_index(show_available_templates=True)
Project Title¶
This is the project's README.md, which is "inlined" in the project index.
If you are directly viewing 'README.md,' see 'project_index.ipynb'
Project Directory: "/home/dinalt/ai_assets/forgather/examples/tutorials/projects_overview"¶
Meta Config¶
Meta Config: /home/dinalt/ai_assets/forgather/examples/tutorials/projects_overview/meta.yaml
Template Search Paths:
Available Configurations¶
Default Configuration: default.yaml
Available Templates¶
The above is the output from nb.display_project_index().
Project Directory¶
This shows the absolute path of the project, which can be useful for opening the project with the notebooks linked at the top of this notebook.
Meta Config¶
This includes a link to the meta-config file (meta.yaml). Clicking on the link will open this file.
This is followed by a template include graph of the meta-config file. This example does not include any other templates, so there is only a single file listed.
Project configurations are can be constructe from Jinja2 templates and the meta-config lists all of the directories which will be searched for templates. This is a simple project, which is entirely self-contained, but most projects will include multiple search paths here.
Available Configurations¶
The meta-config defines where in the template namespace top-level configuration files may be found. This section lists all configurations within this namespace.
This also shows which configuration is the default and which is presently selected. The active template can be specified by filling out the "config_template" argument to nb.display_project_index().
Available Templates¶
(optional) This lists the names of all available configuration templates within the template searchpath.
Configurations¶
A project may contain one or more configurations, named in "Available Configurations" above. If we don't specify a configuration, the default config, named in "meta.yaml," will be selected.
nb.display_config(
config_template="",
show_available_templates=True,
show_pp_config=True,
show_loaded_config=True,
show_generated_code=True,
)
Included Templates¶
Config Metadata:¶
'A simple list'
Modules¶
Output Targets¶
- meta
- main
Preprocessed Config¶
meta: A simple list
main:
- Alpha
- Beta
- Gamma
- Delta
Loaded Configuration¶
meta: 'A simple list'
main:
- 'Alpha'
- 'Beta'
- 'Gamma'
- 'Delta'
Generated Code¶
def construct(
):
return [
'Alpha',
'Beta',
'Gamma',
'Delta',
]
Included Templates¶
Starting with the active configuration, each configuration template is recursively parsed to identity all sub-templates which are directly and indirectly included in the present configuration. This will normally be a sub-set of "Available Templates."
Modules¶
This lists all dynamically imported modules in the project. This project does not use dynamic imports, so the list is empty.
As a side-note, as we do not load any code at this time, this is limited to Python modules which are directly named in the configuration.
Output Targets¶
These are available target objects which can be constructed from the project. See below.
Preprocessed Config¶
All configuration templates use a combination of Jinja2 and YAML syntax. This section shows what the resulting configuration file looks like after being processed by Jinja.
Loaded Configuration¶
After the configuration has been proprocessed and parsed into a graph, the generated graph is translated back into YAML for verification. While this may be identical to the input for trivial configurations, more complex configurations may be refactored and rendered differently than the input.
Generated Source Code¶
A loaded configuration graph can be exported as Python code, which is exactly what this section shows.
The output can itself use a template for formatting, but the project index uses the default code template, which defines a function named 'construct' which return the defined configuration when called.
from forgather.project import Project
import forgather.nb.notebooks as nb
from pprint import pp
# Load default config, "default.yaml"
proj = Project()
# Construct the default target, "main"
main = proj()
# Show the constructed object.
pp(main)
['Alpha', 'Beta', 'Gamma', 'Delta']
# Explicitly specify the configuration to load.
proj = Project("poems.yaml")
# Explicitly construct one or more targets.
meta, main = proj("meta", "main")
pp(meta)
pp(main)
'A list of poems.'
[{'name': 'The Raven', 'author': 'Edgar Allan Po', 'year': 1845},
{'name': 'The Road Not Taken', 'author': 'Robert Frost', 'year': 1915}]
Notebook API Reference¶
def render_project_index(
"""
Render the project index as Markdown and return the string.
"""
project_dir: str = ".",
/,
config_template: str = "",
show_available_templates: bool = False,
) -> str:
...
def display_project_index(
"""
Render the project index as Markdown and display it.
"""
# Same arguments as above.
) -> None:
Likewise, the details of each configuration can be displayed.
def render_config(
project_dir: str = ".",
/,
config_template: str = "",
show_pp_config: bool = False,
show_loaded_config: bool = False,
show_generated_code: bool = False,
pp_first: bool = False,
**kwargs,
):
"""
Render config information
project_dir: The location of the project directory
config_template: The configuration to display. If "", the default is shown.
show_pp_config: Show the preprocessed configuration.
show_loaded_config: After loading the preprocessed configuration, render the node-graph as YAML.
show_generated_code: After loading the preprocessed configuration, render the node-graph as Python code.
pp_first: Normally a config is preprocessed and loaded in a single step. This breaks the process down
into seperate steps, which can be useful for debugging YAML errors.
kwargs: These kwargs are passed in to the Jinja2 environment and are available to templates. Usually not
required.
"""
def display_config(
"""
As above, but display in notebook.
"""
project_dir=".",
/,
**kwargs,
) -> None: