@George Functionality Request: modular modding

[pixelfck]

Hello George,

One of the things that makes Transcendence great is its support for mods. The engine offers everybody options to add new content and overwrite just about anything defined for the Vanilla adventure.

This request is about the second part of this: the overwriting of existing entities.

While the ability to overwrite an entity is very powerful, it is also quite often an overkill for what we want to achieve. If, for example, a mod would like to change the default shield generator for a Wolfen-class gunship, the entire definition for the ship (60-lines of XML) need to be replaced with a near identical copy of the original, just to change a single device.
This has two problems:
1. If two people want to make two non-conflicting changes to the same entity, they will overwrite each other,
2. If the default entity is changed in subsequent releases of Transcendence, the mod will overwrite the new changes because it was based on an old copy of the entity.

So, I would like to propose a syntax that would allow 'modular modding'; to mod an entity, without the need to overwrite the entire declaration.

I've given this quite some thought, and I thought about presenting this idea as a PDF instead, because it would allow me to better highlight examples and give a clearer explanation of the argumentation for design decisions.
Instead I choose to present this idea here, on the forums, because I think it gets more exposure by removing the need to download an external document.

For the 'modular modding' syntax I tried to:
1. make sure it is (self) consistent throughout all functionality,
2. keep it consistent and compatible with the functionality already in Transcendence,
3. adhere to the XML standard (don't make up new meanings or rules),
4. make sure it is as intuitive as possible, for everyone.

[pixelfck]

-- PART I: The XML ------------------------------------------------

Basic syntax for 'modular modding':

Code: Select all

<Element UPDATE="&unid;" />

By specifying the 'UPDATE' attribute instead of the 'UNID', the entity indicates that it will apply changes to the base entity, without replacing it.


Some examples (all these examples are based on the &scWolfen; and &scWolfenPlayer; definitions as can be found in the Transcendence source):


First up is 'modular modding' the attributes, which have the simplest syntax.

Change an attribute:

Code: Select all

  <ShipType UPDATE="&scWolfen;"
          manufacturer="Entrepreneurial Modders"
          />

This would change (overwrite) the 'manufacturer'-attribute from whatever it was into 'Entrepreneurial Modders'.


Change an attributes on a nested element:

Code: Select all

  <ShipType UPDATE="&scWolfen;">
      <Armor armorID="&itDiamondLatticeArmor;" />
      <Armor armorID="&itDiamondLatticeArmor;" />
      <Armor armorID="&itDiamondLatticeArmor;" />
      <Armor armorID="&itDiamondLatticeArmor;" />
  </ShipType>

This would make sure every spawned &scWolfen; is now equipped with 4 segments of diamond lattice armor.


Add an attribute:

Code: Select all

  <ShipType UPDATE="&scWolfen;"
          cyberDefenseLevel="25"
          />

Since there is no 'cyberDefenseLevel'-attribute specified for the default &scWolfen;, this update adds the attribute (syntax is the same as changing/overwriting the attribute, if it was specified).


Remove an attribute:

Code: Select all

  <ShipType UPDATE="&scWolfenPlayer;"
          maxNonWeapons=""
          />

Setting the value of an attribute to an empty string effectively removes it. If you would query (see part II) the &scWolfenPlayer; through scripting, this attribute should be removed from the XML declaration.


Then, 'modular modding' the elements:

Add an element:

Code: Select all

  <ShipType UPDATE="&scWolfen;">
      <Names>The Pacifist</Names>
  </ShipType>

This would name all Wolfen-class gunships 'The Pacifist'.


Add a nested element:

Code: Select all

  <ShipType UPDATE="&scWolfen;">
      <Devices>
	      <Device deviceID="&it50MWReactor;"/>
	  </Devices>
  </ShipType>

This would add a device to all spawned &scWolfen; ships.


Remove an element (and all its children):

Code: Select all

  <ShipType UPDATE="&scWolfen;">
      <Devices />
  </ShipType>

This would completely remove all devices from the ship. If you would query (see part II) the &scWolfen; through scripting, the 'Devices'-element should be removed from the XML declaration, to allow for multiple mods to works as expected.


Replace three nested elements (remove + add)

Code: Select all

  <ShipType UPDATE="&scWolfen;">
      <Devices>
          <Device />
          <Device />
          <Device />
          <Device deviceID="&itSlamCannon;"/>
          <Device deviceID="&itMAGLauncher;"/>
          <Device deviceID="&itHullPlateIonizer;"/>
      </Devices>
  </ShipType>

This declaration first removes the three default devices and then adds three new devices, thereby effectively replacing the original devices.
There is of course no need to replace all devices, mix and match as desired.

[pixelfck]

-- PART II: Scripting ---------------------------------------------

This is where the real fun begins. In order to effectively be able to use scripting to change entities upon game start (OnGlobalTypesInIt-event) or make decisions based for game logic based on the analysis of types, the scripts need to be able to read/analyse the XML declarations.
I can see a whole lot of options combining this with the power of (TypCreate ...), dynamic extending of entities, combining entities into one, etc, etc.


The first function requested is rather simple:

check for UNID to be defined:

Code: Select all

  (unvIsType &scWolfen;) -> True/Nil
  (unvIsType &scWolfen; 'shipType) -> True/Nil

This checks to see if there is an entity defined as &scWolfen;.
The first call return True if such an entity exists for the current game instance,
the second call returns True if the entity exists and is of the requested Type.


The next six requested functions are a lot more complicated. The proposed functions are flexible and powerful. They may be perceived as a bit complicated at first, but they should be rather intuitive when given a second look.

- read attributes, elements and value from type definition:

Code: Select all

  (typGetElements &unid; ...)
  (typGetAttributes &unid; ...)
  (typGetElementValue &unid; ...)

- read the number of attributes, the number of elements and the number of value from type definition:

Code: Select all

  (typGetElementCount &unid; ...)
  (typGetAttributeCount &unid; ...)
  (typGetElementValueCount &unid; ...)

The observant reader will have noted that the argument list has not been included in the six functions listed above, this is because their arguments can best be explained with an example:

An example entity (please note: the XML below would cause an error in Transcendence but that is not the point here, the entity is meant to serve as an illustration only):

Code: Select all

  <SystemMap UNID="&smHumanSpace;"
          displayOn="&smKnownSpace;"
          >
      
      <Node ID="HQ" x="-114" y="313">
          <System UNID=         "&ssBlackMarketHQ;"
                  level=        "8"
                  attributes=   "humanSpace, outerRealm"
                  />
           
           <Names>Ross 248; 5 Indi; Groombridge; Lalande</Names>
      </Node>
	  
      <Stargates>
          <Stargate name="SE" from="SE:Outbound" to="C1:Inbound"/>
          <Stargate name="BA" from="BA:Outbound" to="HQ:Inbound"/>
          <Stargate name="A7" from="A7:Outbound" to="C7:Inbound"/>
      </Stargates>
	  
	  <Comment>Thumbs up!</Comment>
  </SystemMap>

All the function calls below use the XML code illustrated above.

Some simple examples figuring out which elements are present in an entity:

Code: Select all

  (typGetElementCount &smHumanSpace;) -> 3
  (typGetElements &smHumanSpace;) -> (list "Node" "Stargates" "Comment")

The first function call requests the total number of (child) elements on the entity &smHumanSpace;,
The second function call requests the elements themselves.

The same can be done for the (child) element 'Stargates':

Code: Select all

  (typGetElementCount &smHumanSpace; 'Stargates) -> 3
  (typGetElements &smHumanSpace; 'Stargates) -> (list "Stargate" "Stargate" "Stargate")

Then, some elements may contain a value:

Code: Select all

  (typGetElementValue &smHumanSpace; 'Stargates) -> Nil
  (typGetElementValueCount &smHumanSpace; 'Stargates) -> 0
  
  (typGetElementValueCount &smHumanSpace; 'Comment) -> 1
  (typGetElementValue &smHumanSpace; 'Comment) -> "Thumbs up!"

Some elements may contain a list of values:

Code: Select all

  (typGetElementValueCount &smHumanSpace; 'Names) -> 4
  (typGetElementValue &smHumanSpace; 'Names) -> (list "Ross 248" "5 Indi" "Groombridge" "Lalande")

The syntax allows for requesting an index from the list of values:

Code: Select all

  (typGetElementValue &smHumanSpace; 'Names 0) -> "Ross 248"

The syntax also allows for requesting a list of indexes from the list:

Code: Select all

  (typGetElementValue &smHumanSpace; 'Names (list 0 2)) -> (list "Ross 248" "Groombridge")

Having dealt with the elements, the following functions on our list deal with the attributes of entities:

We should of course be able to request the number of attributes for an element:

Code: Select all

  (typGetAttributeCount &smHumanSpace; 'Node) -> 3

Of course, we are more interested in requesting the value of an attribute:

Code: Select all

  (typGetAttributes &smHumanSpace; 'displayOn) -> "0x00200001"
  (typGetAttributes &smHumanSpace; 'Node 'x) -> -114

Some attributes may contain a list of values:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Node 'System 'attributes) -> (list "humanSpace" "outerRealm")

It should be possible to specify a list with the attributes for which we would like to see the values returned:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Node (list 'x 'y))) -> {x: -114 y: 313}

In this case, we request the x="" and y="" attributes from the <Node ...> element. The result should be a struct with the two attributes and their values.


In extension of requesting a list of attributes, it should be possible to request all attributes of an element:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Node) -> {ID: HQ x: -114 y: 313}

In this case, the element has three arguments, all returned in a struct.


A complicating factor in XML is that an element may be specified more than once. In the example above, the element <Stargates> contains three <Stargate ...> elements.
As may be expected, the default syntax returns the attribute(s) from the first <Stargate ...> element:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates 'Stargate 'name) -> "SE"

This is because 'Stargate is a shorthand notation for (list 'Stargate 0). So, in other words, the above example is identical to:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates (list 'Stargate 0) 'name) -> "SE"

Both examples return the attribute 'name' from the first 'Stargate'-element.


Having seen the above, the syntax for requesting the attribute 'name' from the second 'Stargate'-element should be no surprise:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates (list 'Stargate 1) 'name) -> "BA"

Combining the syntax for selecting attributes from the nth element with the syntax for requesting all attributes gives us:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates (list 'Stargate 1)) -> {name: "BA" from: "BA:Outbound" to: "HQ:Inbound"}

There should be no reason why we cannot extend the above into requesting an attribute from the first and the third element:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates (list 'Stargate 0 2) 'from) -> (list "SE:Outbound" "A7:Outbound")

Likewise, we should be able to request all attributes from the first and the third element.

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates (list 'Stargate 0 2)) -> (list {name: "SE" from: "SE:Outbound" to: "C1:Inbound"} {name: "A7" from: "A7:Outbound" to: "C7:Inbound"})

And finally, combining everything above gives us:

Code: Select all

  (typGetAttributes &smHumanSpace; 'Stargates (list 'Stargate 0 2) (list 'from 'to)) -> (list {from: "SE:Outbound" to: "C1:Inbound"} {from: "A7:Outbound" to: "C7:Inbound"})

Looking forward to hear your thoughts,

Cheers,
Pixelfck

[pixelfck]

(reserved)

[FourFire]

This looks very interesting, and I would enjoy being able to use this functionality if implimented.

[george moromisato]

This is really cool--though it may turn out to be difficult to implement. Let me braindump some of the implementation details in hopes of refining the proposal [I'm going to refer to specific bits of code, but it's not necessary to follow the links to get the basic idea.]

The process for dealing with types is roughly as follows:

1. Parse: The first thing we do is parse an XML textfile (a sequence of characters) into a datastructure that represents the XML hierarchy. We end up with a C++ object for each XML element (CXMLElement). Each element object contains child elements and has a list of attribute-value pairs (as strings).

Header: https://github.com/kronosaur/Transcende ... /XMLUtil.h
Implementation: https://github.com/kronosaur/Transcende ... er/XMLUtil

2. Load: Next, for each distinct extension/library in the system, we load the type definitions into specific type objects. For example, a <ShipClass> element gets loaded into a CShipClass object, a <StationType> element becomes a CStationType object, etc. All of these specific type objects are descended from the CDesignType base-class.

These type objects have specific member variables to track parameters of the type. For example, take something like the mass of a ship. In the XML file, this is represented as a sequence of characters:

<ShipClass ...
mass="150"
...

But in the CShipClass object, this becomes an integer member variable:

class CShipClass {
...
private:
int m_iMass;
...
}

There is code in CShipClass that converts the CXMLElement representation to the CShipClass representation.

Why do we do this? For efficiency. We need to access a ship's mass to calculate motion. We do that once per ship 30 times per second. If there are 100 ships in a system, we're accessing that variable 3000 times per second. A quick member-variable access is a lot faster than a hash table lookup (plus potentially a string-to-integer conversion).

Note also that once we load all the types into type objects, we discard the CXMLElement objects. We don't need them anymore, so we don't bother keeping them in memory.

By the time we get to the intro screen, all of the XML has been loaded into type objects and we've discarded all the CXMLElement objects.

Another thing to know is that all overrides are kept in their own extension. For example, imagine that we have a ship class of UNID 0x100 in the core game. Imagine also that an extension overrides UNID 0x100. At load time, we create TWO CShipClass objects, one for the original and one for the override.

Header: https://github.com/kronosaur/Transcende ... SEDesign.h

3. Bind: The last step is to "bind" the types to deal with overloading. Binding happens each time you create or load a game.

The main purpose of binding is to resolve dependencies between types. For example, imagine that station type A creates ship class B (as guards or whatever). Internally, the CStationType object representing A has a pointer to the CShipClass object representing B. But what happens if B gets overridden by an extension?

Remember that there are actually two CShipClass objects representing B. One is the original and the second is the override. At bind-time, we decide which of the two to use (based on whether the extension is loaded or not). And then we set the pointer correctly (either to the original CShipClass or the override).

Lastly, bind-time is also when we generate dynamic types. If we're creating a new game, we execute all the code to generate types and include them either as original types or override. Once we've generated the dynamic types, we can't tell the difference between them and static types.

Implementation: https://github.com/kronosaur/Transcende ... n.cpp#L183

--------------
Where does this leave us? It seems to me that the main implementation obstacle is the part in step 2 where we discard all the original XML data-structures. Perhaps what we need to do is retain the XML objects (and associate them with their type objects) and allow a modder to create dynamic types based on modified versions of the original XML.

The downside to this is memory usage, but perhaps we can do this only if we detect an extension that needs it.

[Atarlost]

If XML elements correspond to data structures we don't need the XML.

We can't alter sub-elements without overriding the whole element because if that is possible then a partial override cannot remove sub-elements. That is, therefore, not a concern.

Attributes of the base item or object I believe become individual variables or structures. If the new ship has a mass value we slap it over the old mass value. If it has a different damage string we parse it and slap it over the old damage structure. This should not be a problem I don't think.

If the new version has different armor we have to replace the whole armor structure. If armor is its own structure.

If the new version has a different inventory we replace the old inventory. If inventory is its own structure.

Events and Staticdata we can fall through since there's a way to remove those in overwriting. But we can already add events to objects by setting ship controllers and staticdata is, like events, ultimately elements containing nothing but strings so there should be no unsolved technical challenges with overriding it.

AFAIK the only hairy bits should be equipment tables since they can include inventory items. Or if inventory and devices or devices and armor share data structures.

[pixelfck]

Where does this leave us? It seems to me that the main implementation obstacle is the part in step 2 where we discard all the original XML data-structures. Perhaps what we need to do is retain the XML objects (and associate them with their type objects) and allow a modder to create dynamic types based on modified versions of the original XML.

The downside to this is memory usage, but perhaps we can do this only if we detect an extension that needs it.

If XML elements correspond to data structures we don't need the XML.

So far, the game engine does (if I'm not mistaken) not supports creating types after game start. I would not expect this to change.

This would mean that, for XML-Updates (part I) only, the engine would need to merge data structures after the player picks a selection of mods and before the bind-step takes place. It may be possible (as Atarlost points out) to merge the c++ objects, so there would be no need to keep the parsed XML data-structures in memory.

However, I don't think that c++ objects have any 'knowledge' of the (nested) structure of the original XML, while the modders cannot be expected to know about the internal data format; they have only the XML-structure to go by.
So, if the scripting (part II) would be implemented, I guess the parsed XML-objects need to be kept around up to after the player has picked a selection of mods and the OnGlobalTypesInIt-event (which could try to query the XML-structure) has completed. Only then the types, included in the player's selection of mods, could be merged into their final incarnation. Followed by the Bind-step to resolve dependencies between types.

I don't think detecting mods that use this feature will really help: if there is just one mod that uses this functionality, the XML-structure for all mods needs to be kept in memory. So, sooner or later, most of the (mod-using) users will have at least one mod that uses the functionality, which makes the detection a mood point I guess.
Also, I don't know what the impact on loading times and memory consumption would be. But it is clear that both will increase. However, if the (initial) game load would exclude any mods, time to start the engine up-to showing the 'menu' screen could probably be speeded up a bit.

Cheers,
Pixelfck

[george moromisato]

The XML objects created in step #1 (CXMLElement) have the full XML hierarchy--they are a lossless representation of the underlying XML files. Modders should be able to manipulate these using the kinds of functions that you propose (typGetElementCount, etc.).

My proposal would be something like this:

1. For each type loaded in step #2, keep the original CXMLElement that it came from.
2. Define a set of functions that can create/modify CXMLElement objects (like in your proposal).
3. Allow defining dynamic types based on the CXMLElement objects created in script.

For now maybe we don't worry about (memory) efficiency.

If we have the above, you could, in theory, create a TLisp library that lets you arbitrarily modify any core XML element.

[digdug]

This is all very cool. Also I really appreciate George's explanation on how the XML is parsed as I can't read C and trying to figure it out by reading the source code by myself would be a futile attempt.

So far, the game engine does (if I'm not mistaken) not supports creating types after game start. I would not expect this to change.

You can definitely typCreate NEW types anytime while the game is running.

You can overwrite types using tyCreate only at game start.

[pixelfck]

@George, Just a quick note on the sideline: your c++ code is really nicely written [at least the parts I've seen so far] which makes it a lot easier for me to read through (and I can't actively program in cpp).

My idea was that any creating/modifying would be done through XML-strings, while reading existing XML would be done through script. This would keep XML part accessible for entry-level modders, while giving more power to the ones who can use TLisp. I figured there would be no need for functions that directly manipulate the existing XML, as it would then overlap with typeCreate + XML-string.

Cheers,
Pixelfck

[Jeoshua]

My feedback on this proposal is thus:

It would be best, if implementing this kind of a thing, to try and look at what other standards there are for things like this. RFC JSON Patching is one really good example.

From http://tools.ietf.org/html/draft-ietf-a ... n-patch-10

Code: Select all

   PATCH /my/data HTTP/1.1
   Host: example.org
   Content-Length: 326
   Content-Type: application/json-patch
   If-Match: "abc123"

   [
     { "op": "test", "path": "/a/b/c", "value": "foo" },
     { "op": "remove", "path": "/a/b/c" },
     { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
     { "op": "replace", "path": "/a/b/c", "value": 42 },
     { "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
     { "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
   ]

Obviously we are not dealing with JSON here, but the principles in this document are all still sound. A proper patch style change would require that the type of change being made, testing or removing, adding, replacing, moving, or copying be specified. The example where one replaces an item would thus be:

Code: Select all

  <ShipType op="modify" UNID="&scWolfen;">
      <Devices>
          <Device op="replace" path="&itLaserCannon;" deviceID="&itSlamCannon;" />
      </Devices>
  </ShipType>

Or something to that effect. You get what I'm saying tho, right?