Jump to content
AL|EN

Updates and Delta Updates for Github mods - brainstorming

Recommended Posts

Hi,

One of the most missed features for all past years was the ability to click one button in order to update my all local mods. Lucky, with the Github API, Project infinity can offer a possibility to update mods, which is very easy to add via one single line inside mod metadata file. Because it's github-based, it offers stability and consistency. It is impossible to download mod with malformed files, you either download it correctly or don't download it all (git feature). And for the first time, there is a possibility for so called "delta updates"!

As a sidenote: Github.com now offers private repositories for free!

Right now, this system is opt-in and it's based on the mod metadata "Download" keyword:

Download=https://github.com/Gibberlings3/SwordCoastStratagems

Wanted features:

  • easy to integrate
  • simple and understandable rules for modders about when mod will be updated and how

Assumptions:

  • Mod use Github hosting (or the site which provide API)
  • Mod has metadata file and filled 'Download=' keyword
  • the update process don't touch mods inside game directory, only the mods from the directory where user has placed all mods

General Flow:

  • Phase 1: Read Mod Metadata  "Download" keyword > Check if update exist > Present 'Update Available' for specific mod > do nothing until player hit "Update Mod" button
  • Phase 2: the "Update Mod" button was clicked > Download update > extract and overwrite all mod folders and files / preform delta update and also overwrite all mod folders and files

How it would work:

  • Checking if there is a updated version of the mod is done automatically (but not the update itself)
  • Notification about the update will be presented to the player ('Update Available' next to the mod name, "Update mod" button enabled")
  • The update itself can be only accepted/started by player himself (in order to not break existing mod/components setup)
  • Leave user-created files inside mod data folder intact for eg: mymod\mymod-config-user.ini

Which update channels would be offered for players:

  • if mod has releases, offer to update only if there is a new release
  • if mod has only prereleases, offer the update for prereleases
  • if mod has both releases and prereleases, offer the update for releases only, do not offer the update for prereleases unless there is a global option "Allow for prereleases" enabled
  • if mod doesn't have any releases or prereleses, the assumption is made (mentioned at 'Download=' keyword documentation ) that the modder use master branch. For many modders, simply committing to master branch seems to be enough and I don't want' to force them to make releases. Modders can simply create Release in order to switch update chanel.

When player decides to update the mod, there are two ways of updating locally extracted mod files:

  • downloading release/prerelease, extract and overwrite all mod folders and files
  • preform delta update, also overwrite all mod folders and files

My main concerns are:

  1. Should I offer "Prereleses" for normal users, who aren't interested testing BETA versions and reporting bugs? (opt-in option?)
  2. Should I offer "download last commit" aka "experimental" for mods which do have releases/prereleases? (undocumented option?)
  3. Should I offer "download last commit" from specified branch like "devel" (undocumented option?) @argent77 AFAIK, you alone use different branches.
  4. What to do with files which were available in older version (let's say MYMod\Items\SWORD.itm) but the new one doesn't have them anymore? Removing old mod version directory entirely could potentially remove also user-generated config (only two mods have them but it's still important). If such problem occur, any kind of solution for is doable at this stage. No worries, files non-present inside new mod version are removed.

That's everything for now, any feedback will be appreciated.

@K4thos Enabling this for EET will allow for EET delta update without re-downloading 250+ MB, what do you think?

Edited by AL|EN

Share this post


Link to post

1. Unless SR and IR (iirc) got new releases, they still use both mechanisms for two different release streams (3 and 4).

2. and 3. You can do this later if there is a need.

4. Are you talking about diffs or binary deltas here? The first don't have this problem.

Share this post


Link to post

@lynx

1.That's why it's opt-in, I'm sure that when SR/IT maintainers will like to have such feature for those mods, a simple rearrange of the repo should not be a problem.

2. and 3. Yep, I can add this later, I'm asking because I've already receive such feature request from one player at beamdog thread.

4. Indeed for diffs it shouldn't be any problem. Binary deltas aren't something which I could achieve without will and direct cooperation with site admins.

Share this post


Link to post

So what kind of deltas are you talking about, just the fact that the mod changed, not actually any format? I don't see a problem there either; things under SCM will update nicely, while for normal archives you can wipe everything but inis and it should also be fine.

Share this post


Link to post

@lynx My main idea is to use git deltas.

After some experimentation, unfortunately it's impossible (or rather only possible if I force update to overwrite altered local mod files which is dangerous operation) for now to support "Delta-Updates" when multiple mods were extracted into the same top-level directory. So while all mods can be scanned and installed regardless of having extra top-level directories, they cannot receive delta-updates unless they have extra top-level directory. 

Share this post


Link to post

My apologies - can you throw a directory structure up that sorts this out for me? And we are talking about user folders/directories for mod archives, not user folders/directories within the game install folder, right?

Share this post


Link to post

@cmorgan Yes, user folders/directories for extracted mods. It has nothing to do with the way how anyone structure his mod. Everything is related to the situation when multiple mods which contain "Download=..." were extracted to the same top-level directory like this:

Quote

ModA
Download=http://github.com/UserA/ModA
D:\ExtractedMods\Quests\ModA\Items
D:\ExtractedMods\Quests\ModA\ModA.tp2
D:\ExtractedMods\Quests\ModA\Setup-ModA.exe
ModB
Download=http://github.com/UserB/ModB
D:\ExtractedMods\Quests\ModB\Lib
D:\ExtractedMods\Quests\ModB\ModB.tp2
D:\ExtractedMods\Quests\ModA\Setup-ModB.exe


both mods has perfectly fine folder structures, they can be copied and installed without problems but they share top-level directory so delta updates can't be preformed safely.

User will have to create one extra top-level directory like this:
 

Quote

D:\ExtractedMods\Quests\ModA\ModA\Items
D:\ExtractedMods\Quests\ModA\ModA\ModA.tp2
D:\ExtractedMods\Quests\ModA\ModA\Setup-ModA.exe
D:\ExtractedMods\Quests\ModB\ModB\Lib
D:\ExtractedMods\Quests\ModB\ModB\ModB.tp2
D:\ExtractedMods\Quests\ModB\ModA\Setup-ModB.exe

in order to be able to preform delta updates.

Edited by AL|EN

Share this post


Link to post

Cool - and thank you. This is pertinent to me; I grep across an archive of mods when troubleshooting (both for my own modding purposes, looking for conflicting or overloaded states touched with I_C_T[1234], and for figuring out mod conflicts).  

We used to stop folks accidentally packaging mods up this way. Sounds like for my archive, it might be a good idea to do this. 

Share this post


Link to post
4 minutes ago, cmorgan said:

We used to stop folks accidentally packaging mods up this way. Sounds like for my archive, it might be a good idea to do this. 

Just to confirm: no changes are required for mods, even if they come with zip without extra top-level directory (good practice anyway). Users will simply create such extra top-level directory by themselves, one time only and all future updates will be handled correctly.

Share this post


Link to post

The download tool could do it for them quite easily.

Also for the deltas, I don't see a problem with shared directories. If two mods have the same filenames, there'll be problems much sooner than this. If you're nuking the dir before updating, then it's not a delta update anymore, but a redownload.

Dealing with user changes is tricky, but you could git stash beforehand and git stash pop afterwards. The only problem is if there are conflicts, since that requires a manual intervention.

Share this post


Link to post
Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...