module load performance between .mod and unpacked.

[bratch9]

I have a large graphic module of imaged, about 10k images when unpacked this comes to about 4GB and packed its about 3.7GB. ( full 7zip unpack time is 48 seconds )

If I have the pack.mod file in modules folder it takes about 25 seconds to load, shown by the console window.

If I unpack the pack.mod file into a folder in the modules folder, then the console windows says it now takes 1.5 seconds to load.

I'm not sure if anything can be done with this, and I'm not sure what you are 'accessing' from the files for information to have to decode info from the zip file. ( Clearly only parts of each file as its quicker than a full unpack. )

Since this is just 'image' files and I do not think you are loading them at this stage, maybe you just need to get the filename and the data from the contents can be deferred until later when you display the image ? ( Maybe you are getting width/height for layout usage ? )

On a different more realistic module, the console window reported 22 seconds to load the .mod, if I strip out the graphics from the .mod and load the .mod it goes down to 8.5 seconds ( for db.xml only. ) and if I unpack the full with graphics ,mod into a folder it now takes about 10 seconds to load. ( So about 12 seconds on just processing the images part of the module. )

Maybe looking into just what is initially 'looked' at on module load for 'image files', a quicker startup can be managed. ( My current campaign takes 125 seconds to boot, getting this a below 60 seconds would be a start. I'm sure other players have larger campaigns that I do. )

[Moon Wizard]

By definition of the compressed file (ZIP) design, the packing/compression of files incurs overhead. You can possibly try using a zip compression tool with different quality/speed settings to see if that will affect performance of unzipping. Unfortunately, that's just a lot of images, and compressing/packing them adds a lot of overhead in the zip/unzip process.

When we are "loading" the modules, our code is already only supposed to be looking at folder/file names when building the asset index, not actually loading the files. I'll have to double-check to make sure it's not loading the images early for some reason. Also, does the mod contains a db.xml file referencing the images? (Not required for image/token packs) If so, that they be an impact as well.

On a related note, do you have the rights to repackage and distribute all 10K of the images?

Regards,
JPG

[bratch9]

Please take a look, as when I try and access the 'assets' -> 'token' or 'image' view I get another 20 seconds 'pause' while it decides to process the directory structure before it displays. ( The assets could be forgiven some time due to actually having to load and display at least the initial 10 or so images it would show in the window. )

Last I checked reading zip 'file/folder' list was very quick, and did not require any unpacking of compressed data, so I don't see how compression methods would changed that. But I can force a 'tar' uncompressed zip version to check if compression is an issue or not. If that gets me to 1.5 seconds like raw directory folder then it will be interesting to know what is happening to access the compressed data. But I suspect it will still be 20+ seconds.

As above, if I have already taken 20+ seconds to 'load' the file structure, then why are you re-doing it on asset window open.. I don't like paying the cost multiple times. I'm sure a modern pc can cope with the directory structure of a few million files, 10k files should be nothing. ( But then you are in c# and lua depending on the mix of storage locations.. )

I do think you need to add a 'timer' for when your winproc update and start with look at stalls of > 10 seconds, because all the ui/game stall in for that period. If we can start with getting these 'long' stalls reduced to less than 10 seconds, then try to getting that to 5 seconds, then 1 second would very much help the feel of the product. ( And will push some items onto threads.. )

As I hit at. if you were pulling filename and image size from the zip pack, when you came to show them in a window or asset view you could then create a blank image texture to display and deferred thread load the image data and ripple load/display then without stalling the main thread to keep responsiveness. I'm sure most user would be very happy seeing the images stream in with a ui that is still responding.

On related note I would not re-package the images, I don't have the rights to do that. I'd provide zip instructions to combine my data with the art pack and create a mod file. The pack only has about 500 images, the other 9k are from different source(s) and I will not be using all those file only a hand full. ( its just nice to not have to keep re-packing the mod file when I want to test a different file, so dumped all the files into my test.mod ) And I'll create a 'file list' so that when the user makes the mod file they only get the required files been moved from the different pack sources into the combined module. But I also plan to talk with the owner of the pack(s), as they might want to do an official release. But for now its a private for my campaign mod.

Thamls,
-pete

[Weissrolf]

You can use Windows' Resource Monitor to analyze whether any files are unpacked during access of the large mod and generally look at what FGU does with file access during those long wait times. My speculation is that the latter is rather CPU bottlenecked than having to do with file-access.

[bratch9]

So a quick test with some files in a 'tokens' folder, definitions.xml, thumbnail.png ( So no client/db.xml )

The tokens folder has 1078 files and takes up 585354 bytes size. ( 3.68MB on disk size due to drive format.. so not that large )

To start the FGU application, it just take the ruleset time to load ( about 15 seconds ), this module does not show up in the module list because of no client/db xml.

Going to 'assets'-> 'tokens' and then double clicking to 'open the bag' for the pack.

zip store -> takes 13 seconds to open
zip ultra deflate -> takes 13 seconds to open
unpacked folder -> takes under a second to open.

If I add an 'art' folder of files to the pack, 12604 files taking up 3.6GB of space

Going to 'assets'-> 'tokens' and then 'open the bag' for the pack.

zip store -> takes 2 mins 35 seconds seconds to open
zip ultra deflate -> takes 2 mins 35 seconds to open
unpacked folder -> takes just over second to open.


So the zip 'storage' format does not seem to cause any real difference.

I'm not sure why the files in the 'art' should add so much extra time to open the 'assets/token' bag folder from the zip ?

In terms of resource, I see next to no disc access (ssd), or serious memory increase (cpu/gpu), but it is flat out on a cpu core.

Again, not sure what it doing, but reading the zip folder/file contents should be quick, and that should be used to 'filter' the files list for the window. And then I would expect it to just load the 10 images that first show up from the filter window. Since in a lot of cases FG just loads art when the 'window' is opened. So I do not see it loading the 1000 token images, or the 12k other images.

But maybe the list management is doing a lot of copy back and forth as its 'expands' the list of files in the system ?

It does look wrong that the contents of a folder it should not be looking into causes such an extra slow down.

[bratch9]

When the zip files gets above 4gb, then FGU does not show the 'bag' to access via 'assets->tokens' ( This is minor, but worth look at as FGU is a 64 bit exe so probably should cope with large files. )

I also did some checks in c++ on zip access and then reading out the name/width/height of the png/jpg files. Which is more work then Moon Wizard says that FG should be doing.

Using the zip, the code can read all 13684 png/jpg names and width/height into a vector list in 427ms.
Doing the same with 'loose' files, takes 1.8 seconds, due to the many open/close file operations and permissions cost for this on windows.

FGU seems to have 'Ionic_zip.unity.dll' so the zip access code is in native code, so should be very close to the c++ version. ( I used static linking of 'libzip' library, while FGU is using a DLL access method. )

I hope Moon Wizard can find a 'silly' why is it doing that issue, but a 2+ minutes stalled FG while it 'looks' at the zip file is way to long.


EDIT: Added a password zip file, so much like the vault zip files.. This takes 761ms to get the name/width/height from the 13684 file. ( zipP.jpg )

[Moon Wizard]

How are you calculating your width/height vector in your example?

JPG

[smelton]

Sent you a DM

[bratch9]

How are you calculating your width/height vector in your example?

JPG

For a PNG, I read the first 24 bytes, and check for the 8 byte PNG header block, and an 'IHDR' header starting at byte 12, then width/height come from the 8 bytes starting at offset 16. ( An older version of PNG does not have the 'IHDR' in this case you can pull the width/height from bytes 8->16. I do not use this in the main.txt file (c++) code. )

For JPG, I read the first 2 bytes for the jpg marker.. then you read the 'blocks' which are 2 byte block codes followed by a 2 byte 'size'. Skip the blocks I do not care about until I find the 'SOF0' or 'SOF1' block header ( 0xffc0, or 0xffc1 ), then read a further 7 bytes and pull the height from offset 3+4 and width from offset 5+6.

Because of the zip compressed/password structure you can not 'skip' about the file so to skip blocks I do not care about I use a 4k buffer that I just dump the non-required block to skip. ( I split it into 4k chunks if its bigger that 4k, which it can be. ) So I only ever read the first linear 'n' bytes until I get to the block with the width/height.

Its not complex, and if you do not need the image data at the time, then just grabbing the width/height and do a minimal data pull from these blocks is quite quick.

( See the previous attached main.txt file, which is the cpp code for the tool I wrote that calls static linked libzip open/readbyte/close functions )

-pete

[bratch9]

I built a pure unity version to test, it required a version of unity that can be selected to .net4.x mode.

Ive renamed the ziptest.cs file to ziptest.txt and the csc.rsp file to csc.txt. ( You just need to select .net4.x mode and add and rename these file into the assets, and apply ziptest.cs to a gameobject to make it call its start function which does the test. )

Obviously its slower than the c++ version, taking about 1.5 seconds to open the zip and scan out the png/jpg width/heights from a partial zip load of the file.

The .net4.x version does not support password protected zip, but then FG using its own c++ dll for zip access which does.

I wanted to make this sample in as pure c# as possible, to see the performance difference.