Generation Gap Pattern

The Generation Gap Pattern is a common way to weave manually written code into generated software artifacts. While this pattern describes how to structure your classes logically it does not teach you where to place these classes physically as files into the filesystem.

Figure 1: The Generation Gap Pattern

Figure 1: The Generation Gap Pattern

Separating generated files from manually written code is best practice for many reasons. It enables you to use version control wisely and makes the distinction between generated and manually written code easy. The Eclipse EMF Modeling Workflow Engine together with M2T Xpand allows you to accomplish this goal in different ways.

In this post, I will present three different approaches that share some common ideas. Each strategy uses a dedicated directory for generated class files whereas manually written subclasses are stored for themselves in a different directory.

General Directory Layout

Both sets of classes belong to the same package but reside in different source folders. Framework classes should be kept in yet another directory and might be placed in a package specific for the framework as seen in Figure 2.

Figure 2: Schematic directory layout of an Eclipse Java project

Figure 2: Schematic directory layout of an Eclipse Java project

In Eclipse you can configure such a project structure by adding new source folders (New → Source Folder…) or by declaring ordinary folders as source folders in the project properties (Properties → Java Build Path → Source).

Please note that folder names such as src-generatedClasses are meant to be schematically and stand for a category of folders each. In real world projects you might have several source folders that contain generated artifacts and you might deal with many folders or even projects that contain manually written classes. In the following sections you will see how these categories of folders can be used in practice.

Generate Once

Peter describes a directory layout many Xtext users are familiar with. Generated artifacts go into a folder src-gen whereas manually written classes will be stored in a folder src-once. Each time you run the generator the content of folder src-gen will be purged and generated again whereas the the content of src-once will remain untouched.

Figure 3: src-once collects one-time generated artefacts for future editing

Figure 3: Generate Once collects one-time generated artifacts for future editing

Since each model element A leads to two classes AbstractA and ConcreteA (Generation Gap Pattern) you can reduce the amount of manual work by generating the class ConcreteA if it does not exist, yet. During the next run of the generator the class ConcreteA won’t be generated again, hence the name “once”.

Configure an additional outlet in your EMF MWE file:

<component class="org.eclipse.emf.mwe.utils.DirectoryCleaner" directory="src-gen"/>
<component class="org.eclipse.xpand2.Generator">
   <metaModel idRef="mm">
   <expand value="MyTamplate::main FOR model">
   <outlet overwrite="true" path="src-gen">
   <outlet name="MANUAL" overwrite="false" path="src-once">
</component>

Using M2T Expand you can than refer to the outlet MANUAL where appropriate

«DEFINE ConcreteClass FOR Element»
   «FILE this.fileName() MANUAL»
   ...
«ENDDEFINE»

This approach works well in many scenarios but has its limits. As seen in Figure 3 problems arise whenever you delete or rename a model element. In these cases each concrete artifact at src-once that does not correspond to a model element anymore is abandoned and will remain with compile errors. This is desired for classes that hold manual code but is cumbersome for those that do not.

Figure 4

Figure 4: Generate Once in package explorer

For scenarios where a huge amount of generated classes is customized only in a few cases the “Generate Once” approach leads to unwanted manual refactoring effort including file movement. Life is getting worse when you are working with a complex folder structure instead of a single flat directory. Figure 4 might give you an idea of how things will look for growing models.

Conditional Generation 1 (with one gen-folder)

The following approach plays well when only a small subset of classes has to be extended with manual code. According to the Generation Gap Pattern for each model element A there is a generated class AbstractA as well as an concrete class ConcreteA. But in contrast to the “Generate Once” approach both files will be placed in the very same folder src-gen and therefore deleted and regenerated for each generator run.
There is one exception to this behavior. Whenever the generator finds a file ConcreteA in the folder src-man it skips the generation process for this particular file. This is how you introduce manual code.

Figure 5

Figure 5: Conditional Generation 1 reduces manual refactoring effort

At first glance this approach seems to be similar to the solution above. But when deleting or moving model elements this approach really pays off. Each concrete class that does not contain manual code will be deleted and regenerated. Therefore, when moving or deleting the model elements no additional manual work has to be done. Also, a quick look at src-man reveals any code that has been written manually.

Figure 6

Figure 6: Conditional Generation 1 in package explorer

To implement the desired behavior of the generator users of MWE can configure an outlet with a special veto strategy. Please note that the outlet MANUAL does not need to keep files (overwrite=”true”), they will be purged by the directory cleaner anyway.

<component class="org.eclipse.emf.mwe.utils.DirectoryCleaner" directory="src-gen"/>
<component class="org.eclipse.xpand2.Generator">
  <metaModel idRef="mm" />
  <expand value="MyTamplate::main FOR model"/>
  <outlet overwrite="true" path="src-gen" />
  <outlet name="MANUAL" overwrite="true" path="src-gen">
    <vetoStrategy class="your.package.CondGenVetoStrategy" path="src-man"/>
  </outlet>
</component>

Such a veto strategy would read like:

public class CondGenVetoStrategy implements VetoStrategy, VetoStrategy2 {

	private String path;

	public void setPath(String value) {
		path = value;
	}

	public String getPath() {
		return path;
	}

	public boolean hasVeto(FileHandle fileHandle) {
		URI fileToBeGenerated = fileHandle.getTargetFile().toURI();
		URI pathToGenerateInto = new File(fileHandle.getOutlet().getPath()).getAbsoluteFile().toURI();
		URI relativeFileName = pathToGenerateInto.relativize(fileToBeGenerated);
		File potentiallyExistingFile = new File(this.getPath(), relativeFileName.toString());

		return potentiallyExistingFile.exists();
	}

	public boolean hasVetoBeforeOpen(FileHandle handle) {
		return hasVeto(handle);
	}
}

oAW 4.x: VetoStrategy2 has been introduced with oAW 4.3.1.

Unfortunately, the advantages of this approach do not come for free: Compared to “Generate Once” it is more difficult to introduce manual code since it is harder to understand the needed steps. One has to create a concrete class in the src-man folder manually (or move it from src-gen and remove all comments that mark them as generated) where no class exists, yet. This might seem to be a no-brainer but can lead into trouble when defining a process for large projects. This is especially true for the unlikely case that you check-in generated code of src-gen.

Conditional Generation 2 (with two gen-folders)

A slighty different way that basically follows the same idea introduces yet another folder src-gencond.

Figure 7

Figure 7: Conditional Genration 2 lists classes that can implemented manually

As you can see in Figure 7 the generated artifacts are split into the folder src-gen where the abstract classes reside and the newly introduced folder src-gencond. The latter contains only the concrete classes that could be replaced by manually implemented classes. In contrast to the “Conditional Generation 1” approach the users are now able to find classes that can be augmented be manual implementations without any challenge. Users still have to move classes from src-gencond to src-man by hand but it is simpler to communicate the necessary steps.

Figure 8

Figure 8: Conditional Generation 2 in package explorer

As you can see in Figure 8 one can easily identify that there are two classes that could be implemented by hand. Such a folder structure can pay off when different groups of developers are dealing with a fairly large and scattered set of classes.

In order to realize “Conditional Generation 2″ you simply reconfigure the outlet MANUAL to write its output to src-gencond. Also, a second directory cleaner removes all files from this directory before code generation starts.

<component class="org.eclipse.emf.mwe.utils.DirectoryCleaner" directory="src-gen"/>
<component class="org.eclipse.emf.mwe.utils.DirectoryCleaner" directory="src-gencond"/>
<component class="org.eclipse.xpand2.Generator">
  <metaModel idRef="mm" />
  <expand value="MyTamplate::main FOR model"/>
  <outlet overwrite="true" path="src-gen" />
  <outlet name="MANUAL" overwrite="true" path="src-gencond">
    <vetoStrategy class="your.package.CondGenVetoStrategy" path="src-man"/>
  </outlet>
</component>

Comparison

Each of these solutions do their job in real world MDSD projects. Where projects with rather stable interfaces go with the “Generate Once” approach larger projects address growing maintenance effort with the “Conditional Generatation 1/2”  solution. The move from “Conditional Generatation 1” towards “Conditional Generatation 2” is rather trival and can smoothly be taken within a running project.

I’d be happy to read about your experience with these or different project setups.

Support my Work

Writing an article like the one you have just read takes me quite an amount of my personal time. Way too often, I invest this time in different interests and decide against another blog post. On the other hand, you can motivate me with your feedback, your thoughts and your ideas. Please leave a comment below or flattr this post if you think it's worth it.

Comments

  1. Karsten Thoms says:

    Hi Heiko! Great that you finally finished this article. You did a real good job here! I will refer to this article often when consulting customers with MDSD best practices.

  2. Peter Friese says:

    Great job, Heiko! BTW, I think your explanation would make for an excellent addition to the oAW Refcard!

  3. ekke says:

    heiko, thanks for this article.
    I didn’t know about vetostrategy before (perhaps not read all docs or news) and already implemented a similar approach by myself.
    of course I’ll switch to the ‘official’ oAW way to do this.
    ekke

  4. Stefano Juri says:

    Hi Heiko,

    we have implemented an adapted version of the generate once pattern. The content of the src-once directory is backuped before the generation process is starded. src-once is then fully cleaned et re-generated. In a third step, the classes that can be found in src-once AND in the backup directory are restored while all other classes are ignored. In this way the ghost classes (ConcreteA in your example) are deleted, the manually customized classes are preserved and the new classes are generated.
    The save and restore processes are implemented as quite simple OAW-Components an can be integrated in a Workflow. The same strategy could be naturally also implemented as an ant build.
    Thank you for your excellent post.
    Stefano

  5. Thanks for your kudos, guys.

    Hi Stefano, I am thinking about a similar approach to overcome problems with generated artifacts under version control where a separate workflow component checks such a “ghost” directory against the newly generated output. This could overcome annoying problems with directories that have to be deleted. Ever implemented anything in that direction?

  6. ekke says:

    Heiko,

    I think there’s a typo:
    should be
    vetostrategy class=”someFullyQualifiedClassName” path=”src-man”
    instead of …src-gen

    there’s anotherone (at least if using oAW 4.3.1):
    should be vetoStrategy
    instead of vetostrategy
    otherwise the workflow doesnt find the add() methode

    ekke

  7. Thank you for that Hint, Ekke. Fixed it.

  8. iWyre says:

    [...] great little piece of third-party software called EOGenerator that implemented something called the generation gap design pattern. Essentially, this little program would create your custom subclasses of EOGenericRecord [...]

  9. [...] great little piece of third-party software called EOGenerator that implemented something called the generation gap design pattern. Essentially, this little program would create your custom subclasses of EOGenericRecord [...]

  10. Albert says:

    Hi Heiko,

    Thank You for this nice article.

    I was wondering how the Generation Gap Pattern works.
    I’m trying to experiment it and separate generated code from non-generated code, but i don’t know how to process.

    Is it possible to upload the test-project above so i can follow the steps.
    I have a problem to configure the workflow and the template to extend the abstract class.

    Greetings,
    Albert

  11. Hey Albert, I am afraid I don’t have that particular project anymore. To start, you could analyze the different projects Xtext’s “New Project Wizard” creates. Those heavily use the Pattern. Compare the type hierarchy of the Modules for example and see how it’s spread across the src-gen and src folder. If you still have open questions, feel free to ask.

  12. Albert says:

    Hey Heiko,

    Thanks for the reply.

    I´ve got it. The solution is pretty cool.
    Thank you again.

    Albert

Leave a Reply