Mod Manager - Merging Specifications

Discuss and distribute tools and methods for modding. Moderator - Grognak
Icehawk78
Posts: 230
Joined: Tue Sep 18, 2012 4:55 pm

Mod Manager - Merging Specifications

Postby Icehawk78 » Wed Oct 10, 2012 4:16 pm

This is an open request for review by anyone interested (other developers and mod creators, specifically) on the working specifications I've formulated for implementing a "mod merging" functionality.

Overview
What I hope to achieve is a method for allowing mutiple mods to safely interact and allow mod developers to reduce the amount of duplicated code in the creation of any mods which modify existing game assets, rather than the wholesale addition of entirely new assets. Complete replacements of entire files and plain additions can be done as already handled in existing managers.

Detailed Proposal
For any file with the extension of *.merge.xml or *.xml.merge in an mod folder, perform the following steps:

  1. Verify that there is a matching original file of the same base name

    Code: Select all

    The mod file 'data/blueprints.merge.xml' would verify the existence of an unpacked 'data/blueprints.xml'
  2. Parse both the base game file and the mod file as XML fragments, optionally correcting any easily fixed errors in the XML format (this is currently needed to parse the base game assets, but may change at a later date).
  3. For each node in the parsed mod XML, look for the attribute 'mergeType'. This may have a value of 'FULL', 'ATTRIBUTES', 'CHILDREN', 'APPEND', or 'NONE'. If the attribute is not present, or is present by the value is either 'NONE' or anything other than the other values above, ignore this node and continue to its next immediate sibling.
  4. Look for the attribute 'mergeMode'. This may have a value of 'TAG' or 'TAG_AND_NAME'. (Optionally, you may also omit this attribute, and assume a value of 'TAG_AND_NAME' for nodes with a 'name' attribute and 'TAG' for nodes without.)
  5. For a mergeMode of 'TAG' do an xpath search in the base XML (from the current level) for the matching tag name and use the first result as the base XML to modify.

    Code: Select all

    <sensors start="false" mergeMode="TAG" />
    would do an xpath search for 'sensors' from the current node
  6. For a mergeMod of 'TAG_AND_NAME' do an xpath search in the base XML (from the current level) for the matching tag name with a name attribute equal to the current node's name attribute and use the first result as the base XML to modify.

    Code: Select all

    <shipBlueprint name="PLAYER_SHIP_HARD" layout="kestral" img="kestral" mergeMode="TAG_AND_NAME" />
    would do an xpath search for 'shipBlueprint[@name='PLAYER_SHIP_HARD']' from the current node
  7. Look for the attribute 'childMode'. This may have a value of 'APPEND', 'MERGE', 'REPLACE', 'DELETE_MATCH', or 'DELETE_ALL'.
  8. If mergeType is 'APPEND' then add the current mod node (and all children) to the current base node and continue to the next mod node's sibling.
  9. If mergeType is 'FULL' or 'ATTRIBUTES' then for each attribute on the mod node that isn't 'mergeType', 'mergeMode' or 'childMode', set the current XML's attribute equal to the mod node's attribute's value.

    Code: Select all

    <sensors start="false" mergeType="ATTRIBUTES"/> which has matched the node
    <sensors power="1" room="3" start="true" img="room_sensors"/>
    would merge as
    <sensors power="1" room="3" start="false" img="room_sensors"/>
  10. If mergeType is not 'FULL' or 'CHILDREN', continue on to the next sibling node.
  11. If childMode is 'APPEND', take the current mod node's children, and append them to the base XML node's children, without removing any nodes.
  12. If childMode is 'DELETE_MATCH', perform the same xpath search that was done for the current node on each immediate child, and remove any nodes which are found that matched.
  13. If childMode is 'DELETE_ALL', simply remove all children from the base XML node and continue to the next sibling node.
  14. If the childMode is 'REPLACE', remove all children from the base XML node, and then add all of the current node's children to the base XML node's children.
  15. If the childMode is 'MERGE', repeat all of the above steps, but using the current node as the new 'mod node' and the current matched base XML as the new base XML node to perform further searches on. (See detailed example below.)
  16. Continue to the next sibling node.

This may be a bit confusing for those of you who aren't already very familiar with XML and programming, so I've also included a sample mod before/after below:

blueprints.merge.xml:

Code: Select all

<crewBlueprint name="randomizer" mergeType="APPEND">
  <desc>Humans are common and uninteresting. This human has a disregard for control.</desc>
  <cost>40</cost>
  <bp>2</bp>
  <title>Randomizer</title>
   <short>Randomizer</short>
   <rarity>0</rarity>
   <powerList>
      <power>No exceptional traits</power>
   </powerList>
</crewBlueprint>

<shipBlueprint name="PLAYER_SHIP_HARD" layout="kestral" img="kestral" mergeType="CHILDREN" childMode="MERGE">
   <class mergeType="CHILDREN" childMode="REPLACE">Randomizer</class>
   <name mergeType="CHILDREN" childMode="REPLACE">The Randomizer</name>
   <desc mergeType="CHILDREN" childMode="REPLACE">This ship is not for the faint of heart. Your only guarantees are a single pilot and the absolutely bare minimum to get the ship out of the loading bay. Ship equipment will be randomized upon starting.</desc>
   <systemList mergeType="CHILDREN" childMode="MERGE">
      <sensors start="false" mergeType="ATTRIBUTES"/>
      <medbay start="false" mergeType="ATTRIBUTES"/>
      <shields start="false" mergeType="ATTRIBUTES"/>
   <weapons start="false" mergeType="ATTRIBUTES"/>
   </systemList>
   <weaponSlots mergeType="CHILDREN" childMode="REPLACE">3</weaponSlots>
   <droneSlots mergeType="CHILDREN" childMode="REPLACE>3</droneSlots>
   <weaponList count="0" missiles="0" mergeType="FULL" childMode="DELETE_ALL"/>
   <crewCount amount="1" class="randomizer" mergeType="ATTRIBUTES"/>
</shipBlueprint>


blueprints.xml (snippet):

Code: Select all

<shipBlueprint name="PLAYER_SHIP_HARD" layout="kestral" img="kestral">
   <class>Kestrel Cruiser</class>
   <name>The Kestrel</name>
   <desc>This class of ship was decommissioned from Federation service years ago.  After a number of refits and updating this classic ship is ready for battle.</desc>
   <systemList>
      <pilot power="1" room="0" start="true" img="room_pilot">
         <slot>
            <direction>right</direction>
            <number>0</number>
         </slot>
      </pilot>
      <doors power="1" room="2" start="true" img="room_doors"/>
      <sensors power="1" room="3" start="true" img="room_sensors"/>
      <medbay power="1" room="4" start="true" img="room_medbay">
         <slot>
            <number>1</number>
         </slot>
      </medbay>
      <oxygen power="1"  room="13" start="true" img="room_oxygen"/>
      <shields power="2" room="5" start="true" img="room_shields"/>
      <engines power="2" room="14" start="true" img="room_engines"/>
      <weapons power="3" room="10" start="true" img="room_weapons"/>
      <drones power="2" room="1" start="false"/>
      <teleporter power="1" room="15"   start="false"/>
      <cloaking power="1" room="8" start="false"/>
   </systemList>
   <weaponSlots>4</weaponSlots>
   <droneSlots>2</droneSlots>
   <weaponList count="2" missiles="8">
      <weapon name="MISSILES_2_PLAYER"/>
      <weapon name="LASER_BURST_3"/>
   </weaponList>
   <health amount="30"/>
   <maxPower amount ="8"/>
   <crewCount amount = "3" class="human"/>
</shipBlueprint>


resulting merge (snippet):

Code: Select all

<shipBlueprint name="PLAYER_SHIP_HARD" layout="kestral" img="kestral">
  <class>Randomizer</class>
  <name>The Randomizer</name>
  <desc>This ship is not for the faint of heart. Your only guarantees are a single pilot and the absolutely bare minimum to get the ship out of the loading bay. Ship equipment will be randomized upon starting.</desc>
  <systemList>
    <pilot power="1" room="0" start="true" img="room_pilot">
      <slot>
        <direction>right</direction>
        <number>0</number>
      </slot>
    </pilot>
    <doors power="1" room="2" start="true" img="room_doors"/>
    <sensors power="1" room="3" start="false" img="room_sensors"/>
    <medbay power="1" room="4" start="false" img="room_medbay">
      <slot>
        <number>1</number>
      </slot>
    </medbay>
    <oxygen power="1" room="13" start="true" img="room_oxygen"/>
    <shields power="2" room="5" start="false" img="room_shields"/>
    <engines power="2" room="14" start="true" img="room_engines"/>
    <weapons power="3" room="10" start="false" img="room_weapons"/>
    <drones power="2" room="1" start="false"/>
    <teleporter power="1" room="15" start="false"/>
    <cloaking power="1" room="8" start="false"/>
  </systemList>
  <weaponSlots>3</weaponSlots>
  <droneSlots>3</droneSlots>
  <weaponList count="0" missiles="0" />
  <health amount="30"/>
  <maxPower amount="8"/>
  <crewCount amount="1" class="randomizer"/>
</shipBlueprint>
<crewBlueprint name="randomizer" mergeType="APPEND">
  <desc>Humans are common and uninteresting. This human has a disregard for control.</desc>
  <cost>40</cost>
  <bp>2</bp>
  <title>Randomizer</title>
  <short>Randomizer</short>
  <rarity>0</rarity>
  <powerList>
    <power>No exceptional traits</power>
  </powerList>
</crewBlueprint>


Thoughts? Questions? Suggestions?
ComaToes
Posts: 13
Joined: Sun Sep 23, 2012 9:54 pm

Re: Mod Manager - Merging Specifications

Postby ComaToes » Wed Oct 10, 2012 11:19 pm

Whilst thorough, your proposal would require a fair bit of effort to implement in mod editors and managers.

I would suggest using diff and patch which have been doing this job for years. There are implementations available in various languages and command line versions for most platforms if you can't find a library that suits your needs.
Icehawk78
Posts: 230
Joined: Tue Sep 18, 2012 4:55 pm

Re: Mod Manager - Merging Specifications

Postby Icehawk78 » Thu Oct 11, 2012 1:43 am

ComaToes wrote:Whilst thorough, your proposal would require a fair bit of effort to implement in mod editors and managers.

I would suggest using diff and patch which have been doing this job for years. There are implementations available in various languages and command line versions for most platforms if you can't find a library that suits your needs.

Currently, to the best of my knowledge, there aren't any mod editors yet (besides, y'know, a text editor), and I'd guess that my method is easier to implement by hand for mod creators currently unfamiliar with diff, as well as being naively easier to do (ie you don't have to make changes to the base file and keep an original vs modified version). For any mod editors that are created, I'd imagine either method would be equally easy to implement. (This could be biased due to familiarity now, though.)

For mod managers, I can guarantee that my method is at least partially easier to implement because... I already did. :P I'm currently in the process of getting the rest of the "manager" functions (ie unpacking data.dat, looping over files, implementing a GUI, etc) at which point once it's "working" I'll upload to github (current estimate: tomorrow afternoon).