XML modification tutorial

This is a moderated forum that collects tutorials, guides, and references for creating Transcendence extensions and scripts.
Post Reply
NMS
Militia Captain
Militia Captain
Posts: 569
Joined: Tue Mar 05, 2013 8:26 am

Have you ever wanted to make a mod that makes a small change to an existing type? If you read the Modding Reference section, I'm sure you have. Overriding the entire type may mean including huge amounts of code from the core game with only slight changes, and makes your mod incompatible with other mods that override the same type. Type overrides like <ShipClassOverride> avoid these problems, but can only make certain changes to certain types and are not well documented. XML functions to the rescue!

Here's George's post introducing these functions.
George Moromisato wrote:The basic recipe looks like this:

1. In the root element of your extension (<TranscendenceExtension>), add usesXML="true". This tells the engine that you need the XML for all the types to be kept around for you.
2. In <OnGlobalTypesInit>, you may use a new function, typGetXML to get the XML for a type that you wish to override.
3. There are new functions that manipulate the XML returned by typGetXML. For example, you can set attributes or add elements.
4. Once you've manipulated the XML, you may call typCreate to dynamically create a new type or override an existing type.
Here are the XML functions:

Code: Select all

(xmlAppendSubElement xml xmlToAdd [index]) -> True/Nil
(xmlAppendText xml text [index]) -> True/Nil
(xmlCreate xml) -> xml
(xmlDeleteSubElement xml index) -> True/Nil
(xmlGetAttrib xml attrib) -> value
(xmlGetAttribList xml) -> list of attribs
(xmlGetSubElement xml tag|index) -> xml
(xmlGetSubElementCount xml) -> number of sub-elements
(xmlGetSubElementList xml [tag]) -> list of xml
(xmlGetText xml [index]) -> text
(xmlGetTag xml) -> tag
(xmlSetAttrib xml attrib value) -> value
(xmlSetText xml text [index]) -> True/Nil

Now I'll demonstrate with a slightly more complicated example than George's. I'll use a request from relanat:
relanat wrote:
Fri Mar 10, 2017 1:01 am
Is there a way to add <Trade> info to a station without overwriting the whole station code?
...
I would like to add

Code: Select all

	<SellShip	criteria="s L:1-13;" priceAdj="110"/>
without having to add the nearly 2000 lines of Korolov code to a mod to overwrite it.

So, following George's steps:

Step 1: Make sure your mod has usesXML="true". As of API 37, this is no longer necessary.
Also, if it only affects types in a specific library, you should probably extend that library, so it can't be included in games where it won't do anything. You may also want to include the library so you can use UNIDs from it without declaring <!ENTITY>s for them. In this case, Korolov is definitely part of the Human Space Library, so I'll use that:

Code: Select all

<TranscendenceExtension UNID="&unidKorolovXMLDemo;"
	name=			"Korolov XML demo"
	extends=		"&unidHumanSpaceLibrary;"
	usesXML=		"true"
	apiVersion=		"35"
	version=		"1.0"
	credits=		"Nathaniel Stalberg (NMS)"
>

	<Library UNID="&unidHumanSpaceLibrary;"/>
Step 2: Create an <OnGlobalTypesInit> event and get the XML of the type you want to modify.
This event is the only time you can change the XML of a type that's already in the game. So the changes will only take effect when starting a new game. However, you can get the XML of types and create new types at other times. (But you can't get the XML of a type that has an override type affecting it after this event. Also, there used to be a bug preventing you from getting the XML of a type that had been created or modified with typCreate, but this was fixed in the 1.7 betas.) In this case I'm going to add a generic <Type> to contain the event, but you can also put it inside another type:

Code: Select all

	<Type UNID="&evAddKorolovShipSales;">

		<Events>
			<OnGlobalTypesInit>
				(block (KorolovXML ...)
						(setq KorolovXML (typGetXML &stKorolovShipping;))
Step 3: Modify the XML.
Since I want to add an element that doesn't already exist, I have to specify it somehow. There are a few options:

- Option A: Just include the XML elements you want to add in one of your own types.
This is the easiest option when you know exactly what you want to add. For instance, I can put relanat's <SellShip> element in my <Type>, after <Events>. This is usually fine, but if what you're adding might actually do something in the type you're adding it to, enclose it in another element that won't do anything, like <XMLToAdd>.

Code: Select all

(setq subelementToAdd (xmlGetSubelement (typGetXML &evAddKorolovShipSales;) 'SellShip))
- Option B: Create the XML from a string with (xmlCreate).
This is the trickiest to format because the string is going to be parsed in several different steps. The < and > symbols in the string you want to become XML must be replaced by &lt; and &gt; or your mod will not be valid XML. Also, " symbols must be escaped by putting a backslash before them, like this: \", otherwise the TransLisp compiler will think the string ends and you'll get an error, which could be about mismatched quotes, mismatched parenthesis, unknown entity, etc.

Code: Select all

(setq subelementToAdd (xmlCreate "&lt;SellShip criteria=\"s L:1-13;\" priceAdj=\"110\"/&gt;"))
- Option C: Assemble the XML with XML functions.
You still have to add any new tags like in option A or B, but you can change their attributes and text, and add subelements as I'll do below. Recommended if you want the XML to vary depending on other code.

Code: Select all

(setq subelementToAdd (xmlCreate "&lt;SellShip/&gt;"))
(xmlSetAttrib subelementToAdd 'criteria "s L:1-13;")
(xmlSetAttrib subelementToAdd 'priceAdj '110)
- Option D: Use a CDATA block.
This has the advantage that you can include <, >, and ", but the disadvantage that you can't use replacements beginning with &, including text UNIDs like &unid;. I don't recommend it unless you know what you're doing. See this thread.

Now it's time to add the new element to the station type's <Trade> element:

Code: Select all

(setq tradeSubelement (xmlGetSubelement KorolovXML 'Trade))
(xmlAppendSubelement tradeSubelement subelementToAdd)
Note that (xmlGetSubelement) doesn't just return the subelement; it's more like a pointer to the subelement as part of the larger block of XML. So modifying tradeSubelement also modifies KorlovXML. But it doesn't actually change the XML of the type, so we need to...

Step 4: Override the type with (typCreate).
Easy enough:

Code: Select all

(typCreate &stKorolovShipping; KorolovXML)
Putting it all together, using option A:

Code: Select all

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE TranscendenceExtension [
	<!ENTITY unidHumanSpaceLibrary			"0x00100000">
	<!ENTITY unidKorolovXMLDemo		    	"0xE127B100">
	<!ENTITY evAddKorolovShipSales		    "0xE127B101">
]>

<TranscendenceExtension UNID="&unidKorolovXMLDemo;"
	name=			"Korolov XML demo"
	extends=		"&unidHumanSpaceLibrary;"
	usesXML=		"true"
	apiVersion=		"35"
	version=		"1.0"
	credits=		"Nathaniel Stalberg (NMS)"
>

	<Library UNID="&unidHumanSpaceLibrary;"/>

	<Type UNID="&evAddKorolovShipSales;">

		<Events>
			<OnGlobalTypesInit>
				(block (KorolovXML  tradeSubelement subelementToAdd)
					(setq KorolovXML (typGetXML &stKorolovShipping;))
					(setq subelementToAdd (xmlGetSubelement (typGetXML &evAddKorolovShipSales;) 'SellShip))
					(setq tradeSubelement (xmlGetSubelement KorolovXML 'Trade))
					(xmlAppendSubelement tradeSubelement subelementToAdd)
					(typCreate &stKorolovShipping; KorolovXML)
				)
			</OnGlobalTypesInit>
		</Events>
		
		<SellShip	criteria="s L:1-13;" priceAdj="110"/>
		
	</Type>
</TranscendenceExtension>
Now, you may notice that some of the variables are only used once. I've used more variables and more lines than necessary to improve clarity. The event could be condensed to this:

Code: Select all

(block ((KorolovXML  (typGetXML &stKorolovShipping;))) ; step 2
	(xmlAppendSubelement (xmlGetSubelement KorolovXML 'Trade) (xmlGetSubelement (typGetXML &evAddKorolovShipSales;) 'SellShip)) ; step 3
	(typCreate &stKorolovShipping; KorolovXML) ; step 4
)
But it can't be condensed any further because some xml functions, including xmlAppendSubelement, don't return the result. So you have to store the data being modified in a variable.

Coming soon: Deleting subelements and safely handling types that may not exist, may not have gettable XML, and may not have the expected subelements.
Last edited by NMS on Thu May 24, 2018 10:43 am, edited 3 times in total.
User avatar
digdug
Fleet Admiral
Fleet Admiral
Posts: 2620
Joined: Mon Oct 29, 2007 9:23 pm
Location: Decoding hieroglyphics on Tan-Ru-Dorem

great example !
This is really good !
NMS
Militia Captain
Militia Captain
Posts: 569
Joined: Tue Mar 05, 2013 8:26 am

Thanks! OK, advanced topics:

Deleting a subelement by name
The (xmlDeleteSubelement) function doesn't accept a tag, only an index. So you have to find the index of the element you want to delete. For instance, suppose that the <Trade> element in the Korolov example might already have a <SellShip> element, and you want to make sure yours replaces it. You could do something like this:

Code: Select all

(for index 0 (subtract (xmlGetSubelementCount tradeSubelement) 1)
	(if (eq (xmlGetTag (xmlGetSubelement tradeSubelement index)) 'SellShip)
		(setq subelementToDelete index)
	)
)
(if subelementToDelete (xmlDeleteSubelement tradeSubelement subelementToDelete))
You don't want to delete a subelement while you're iterating forwards because that will change the indices of the subelements after it, so this remembers the index, then deletes it. However, this code will only delete the last matching subelement. If there might be more than one and you want to delete all of them, you can do something like this:

Code: Select all

(setq index (subtract (xmlGetSubelementCount tradeSubelement) 1))
(loop (geq index 0)
	(block nil
		(if (eq (xmlGetTag (xmlGetSubelement tradeSubelement index)) 'SellShip)
			(xmlDeleteSubelement tradeSubelement index)
		)
		(setq index (subtract index 1))
	)
)
In this case I'm iterating backwards, so deleting subelements won't affect the ones I haven't gotten to yet.

Safety (avoiding errors and crashes)
If you attempt to get the XML of a type that doesn't exist, or pass something that isn't a valid XML block to an XML function, you'll get an error. This can crash the game, depending on where the code is. The mod above is pretty safe because it only modifies stKorolovShipping, which is know to exist in the library that it extends. But what if the mod also did other things and you wanted to be able to include it in games where that type might not exist?

The safe thing to do is to check that getting the XML is not an error and that the XML is not nil:

Code: Select all

(if (and
		(not (isError (typGetXML &stKorolovShipping;)))
		(typGetXML &stKorolovShipping;)
	)
	(block ... ) ; OK to make changes
	; not OK to make changes
)
There's one other issue. What if the type and its XML exist, but the <Trade> subelement doesn't (perhaps because another mod removed it)? In this case we want to add that subelement:

Code: Select all

(if (not (xmlGetSubelement KorolovXML 'Trade))
	(xmlAppendSubelement KorolovXML (xmlCreate "&lt;Trade/&gt;"))
)
(setq tradeSubelement (xmlGetSubelement KorolovXML 'Trade))
So with the code to remove existing <SellShip> elements and these safety features, the event in the mod above becomes:

Code: Select all

(if (and
		; Make sure the type exists and XML is not nil.
		(not (isError (typGetXML &stKorolovShipping;)))
		(typGetXML &stKorolovShipping;)
	)
	(block (KorolovXML  tradeSubelement subelementToAdd index)
	
		; Get the XML.
		(setq KorolovXML (typGetXML &stKorolovShipping;))
		
		; Get the new SellShip element (this should be safe because it's in our own mod).
		(setq subelementToAdd (xmlGetSubelement (typGetXML &evAddKorolovShipSales;) 'SellShip))
		
		; Make sure the Trade subelement exists.
		(if (not (xmlGetSubelement KorolovXML 'Trade))
			(xmlAppendSubelement KorolovXML (xmlCreate "&lt;Trade/&gt;"))
		)
		(setq tradeSubelement (xmlGetSubelement KorolovXML 'Trade))
		
		; Delete any SellShip elements that already exist.
		(setq index (subtract (xmlGetSubelementCount tradeSubelement) 1))
		(loop (geq index 0)
			(block nil
				(if (eq (xmlGetTag (xmlGetSubelement tradeSubelement index)) 'SellShip)
					(xmlDeleteSubelement tradeSubelement index)
				)
				(setq index (subtract index 1))
			)
		)
		
		; Add the new SellShip element.
		(xmlAppendSubelement tradeSubelement subelementToAdd)
		
		; Override the station type.
		(typCreate &stKorolovShipping; KorolovXML)
	)
	; You could print an error message to the log here if you expect the type to exist, but it doesn't.
)
Edit: I mistakenly thought (while) was a function. That should be (loop).
relanat
Militia Captain
Militia Captain
Posts: 941
Joined: Tue Nov 05, 2013 9:56 am

IMPORTANT EDIT: I put a typo in my request topic which was inadvertantly carried over to this topic.

Code: Select all

<SellShip criteria="s L:1-13;" priceAdj="110"/>
should NOT have a semi-colon ; after the 's' criteria. The correct code is that shown above.
==================

This is excellent. Many thanks for doing this. It has opened up a heap of options for different mods.

Some clarification if possible.

Which are the subelements or does it vary? <Trade> is one, but is <SellShip> one as well or is it a tag because it is inside the <Trade> subelement?

In that case <SellShip> would be a subelement in your addXML mod but would change to a tag when inserted into the Korolov station code and therefore be inside <Trade>. Is that right?

Or possibly they are all subelements?

Separately;

If I understand correctly we can't 'typGetXML' the Korolov station XML again after the <OnGlobalTypesInit> event but if Option C was used the attributes inside the added XML can be changed. Or can they be changed regardless of the Option used? And can the original unchanged XML still have its attributes changed?

As an example, ships available could be limited based on system level by using

Code: Select all

(xmlSetAttrib 'sellShip 'criteria (cat "s L:" (sysGetLevel) "-13;")) ;or something similar
instead of using code to reduce the size of the available ship list.
(Note: <SellShip> is used to determine which ships are available through RPGShipBroker.xml which isn't currently used in SOTP although the file is included.)
Stupid code. Do what I want, not what I typed in!
NMS
Militia Captain
Militia Captain
Posts: 569
Joined: Tue Mar 05, 2013 8:26 am

relanat wrote:
Tue Mar 14, 2017 11:26 pm
IMPORTANT EDIT: I put a typo in my request topic which was inadvertantly carried over to this topic.
I corrected the string in the first post.
relanat wrote:
Tue Mar 14, 2017 11:26 pm
Some clarification if possible.
"Tag" is the term for the name of an XML element. Any element that's inside another element is considered a subelement of that element. So, for instance

Code: Select all

(if (eq (xmlGetTag (xmlGetSubelement tradeSubelement index)) 'SellShip)
checks to see whether the specified subelement of tradeSubelement is a <SellShip> element.
relanat wrote:
Tue Mar 14, 2017 11:26 pm
If I understand correctly we can't 'typGetXML' the Korolov station XML again after the <OnGlobalTypesInit> event but if Option C was used the attributes inside the added XML can be changed. Or can they be changed regardless of the Option used? And can the original unchanged XML still have its attributes changed?
You can use (typGetXML) after <OnGlobalTypesInit>, except for types that have been modified by a type override element (this mostly affects ship classes that have HD versions added by Stars of the Pilgrim HD or PlayerShip Drones). However, you cannot call (xmlCreate) on a type that already exists after that event. The XML for a type is locked in when it's created, which is at the start of the game for types that are defined in files. So you can't change a station definition system by system. You could create a new version of the station for each system and replace the normal ones with your versions, but that would break things that search for it by type.

With items, you can use a level curve when generating the available items, and you can use (objAddSellOrder) to change what items a specific object sells. These options don't seem to exist for ship sales, but you could ask George to add them.

You can use a mix of my suggested methods to generate the XML when creating or modifying types.
relanat
Militia Captain
Militia Captain
Posts: 941
Joined: Tue Nov 05, 2013 9:56 am

Excellent. Thank you again. Got it now. This makes it clearer too. https://ministry.kronosaur.com/record.hexm?id=69815

Couple of things.
It seems that any library you want to extend has to be in hexUNID format, not text/entity format. So

Code: Select all

extends= "0x00100000"
not &unidHumanSpaceLibrary;. The textUNID gives an "Invalid entity: unidHumanSpaceLibrary" error.

Possibly (take anything I say with a grain of salt because I haven't got enought time to double check everything) you can only add one element at a time. Trying to add both lines of <XMLToAdd> at once didn't seem to work but individually, as in the example code and attached file, did.

Also I'm getting the same "needs to close" error message with this code as with digdug's Shileld Capacitors. http://forums.kronosaur.com/viewtopic.php?f=24&t=7791 It seems to only occurs if the game has started, ie the ship is on the screen. Not partway through the code when the game is loading. This suggests to me that typCreate is doing it somehow. Given that I've seen this twice now, is it worth a Ministry ticket?

Here's the code:

Code: Select all

<ItemType UNID="&vtD789AddShipBrokerToArcology;"
	virtual=	"true"
	>

	<XMLToAdd>
		<SellShip criteria="s L:1-13;" priceAdj="110"/>
		<BuyShip criteria="s L:1-13;" priceAdj="90"/>
	</XMLToAdd>

	<Events>
		<OnGlobalTypesInit>
			(block Nil
				(setq stKatsXML (typGetXML 0x0010201F))
				(setq tradeSubelement (xmlGetSubelement stKatsXML 'trade))
				(setq allXML (typGetXML &vtD789AddShipBrokerToArcology;))
				(setq allAddXML (xmlGetSubelement allXML 'XMLToAdd))
				(setq subelementToAdd (xmlGetSubelement allAddXML 'sellShip))
				(xmlAppendSubelement tradeSubelement subelementToAdd)
				(setq subelementToAdd2 (xmlGetSubelement allAddXML 'buyShip))
				(xmlAppendSubelement tradeSubelement subelementToAdd2)

				(typCreate &stStKatsArcology; stKatsXML)
			)
		</OnGlobalTypesInit>
See the attached mod for the rest of it. It also adds a scrAddAction to the Arcology so this was a convenient <Type> to place it in.
Random stuff.
Put the code outside of <OnGlobalPaneInit>. I was having trouble accessing it when inside. Also enclosing it in <XMLToAdd> may not have been necessary but my modding is a case of trying stuff until it works, and this worked. I think you also need both <SellShip> and <BuyShip> XML in the station code for ShipBroker.xml to work, something to do with calculating sell and buy prices in the code even if you only want to sell ships.

A working SOTP mod is attached which lets you change playerships at St Kats (after all this I'll do something else at the Korolov station :lol:)
Use game version 1.7 because I don't know when all the xml functions were introduced.

Thanks again, NMS. IMO this is exactly the type of documentation that is needed. Well done.
Attachments
StKatsShipBroker.xml
(1.81 KiB) Downloaded 362 times
Stupid code. Do what I want, not what I typed in!
Post Reply