Map load times and map/3dobject compression

[Learner]

Map load times have been improved in the current CVS without any compile options.

 

Also, there is an optional map & 3D object compression that has recently been checked in that requires -DZLIB and ( -lz or -lzlib ) if not already being linked in. With this option enabled you can gzip compress all .elm and .e3d files (it's pk to leave the extensions of .elm.gz & .e3d.gz).

 

While most windows users won't have gzip available, they can use the gzip compressed files from Linux usiers.

 

Please test & post any comments on both of these speed ups.

[LabRat]

I have tried the new CVS and I have to say that the loading times have decreased dramatically, from an estimated 6-7 seconds for the WS map originally to less than 2 with .e3d.gz and .elm.gz

 

Can this thread be deleted and the devs keep this for ourselves?

 

Here is what happened to the directory sizes:

Vanilla 1.3.3 134 MB (141,468,659 bytes)

gzip+libs+cvs 59.8 MB (62,769,155 bytes)

 

gzip+libs+cvs installer (just a rar.exe) 37.4 MB (39,279,033 bytes)

vanilla installer 29.1 MB (30,573,459 bytes)

 

more than a 50% reduction in directory size just using the .gz seems great to me, couple that with a massive load improvement and we have a definite winner \o/

[Kindar_Naar]

more than a 50% reduction in directory size just using the .gz seems great to me, couple that with a massive load improvement and we have a definite winner \o/

 

Yup, I agree. I haven't done any benchmarks, but the improvement in load speed is clearly visible Great work

[crusadingknight]

With so many e3ds to load, and most being within the size of a single DMA push, the major slowdown for them should be seek time - blobbing them into a common random-access archive format along with their textures might be better than having to do a seek for each one, since few are large enough anyway that individual compression helps.

 

Really great speedups though, I must say, most of all in the .elm department.

Edited February 11, 2007 by crusadingknight

[Learner]

With so many e3ds to load, and most being within the size of a single DMA push, the major slowdown for them should be seek time - blobbing them into a common random-access archive format along with their textures might be better than having to do a seek for each one, since few are large enough anyway that individual compression helps.

 

Really great speedups though, I must say, most of all in the .elm department.

some E3D's are huge, but most of them are small. The patch allows reading either gzipped or ungzipped files easily. But another place where it wins is that the OS caching can now store more files as well.

 

Yes, blobbing them into a single file or group of files would help a little more, but not needed currently, maybe later

[bluap]

Nice change A few quick tests suggests it could be a ~30% reduction for me for initially loading the game and something similar for moving from IP->WS the first time

[Espresso_Boy]

Without any quantitative testing, I'll just say it feels leaps and bounds faster.

[Learner]

Who other then LabRat has also taken the time to gzip their files? Are you noticing additional speedup with the compressed files also?

 

Everyone can benefit from the delayed texture loading, but more feedback on the compressed files portion would be good.

[Kindar_Naar]

I run the client only with gzipped files.... and after seeing the speed up I'm not going back to regular ones

[Pardu]

compressing files works great!.. I change maps so fast its like being teleported...thank you..and since I usually only post with whining and problems... another .. thank you .. and keep up the great work

[Learner]

I also checked in support for all .bmp's being able to be gzipped except texturs/cursor.bmp. I don't expect this to affect performance, but it makes it more consistant and can improve OS cache hit rates on system with low memory.

[bluap]

Who other then LabRat has also taken the time to gzip their files? Are you noticing additional speedup with the compressed files also?

I'm only on Linux. I gziped all my .elm and e3d files. I didn't speed test the new code without the gziped files, only "1.3.3 released binary and unziped files" against "latest cvs code and gziped files". I'm assuming I would not otherwise notice the difference of a native build against the released binary.

[LabRat]

If I read correctly, even without gzipping anything you will still see a difference due to stage one - loading only when an item is needed rather than at map load.

 

Next stage was gzipping the .elm and .e3d files

 

After that was bitmap compression - I couldn't really see a speed difference but I have no doubt there was one there, however miniscule.

 

I compressed my exe and dll files with an exe compressor while I was at it (not pertinent to the discussion but I just thought I'd mention it )

[Learner]

If I read correctly, even without gzipping anything you will still see a difference due to stage one - loading only when an item is needed rather than at map load.

 

Next stage was gzipping the .elm and .e3d files

 

After that was bitmap compression - I couldn't really see a speed difference but I have no doubt there was one there, however miniscule.

 

I compressed my exe and dll files with an exe compressor while I was at it (not pertinent to the discussion but I just thought I'd mention it )

Correct.

 

Another side effect of the compression is that in the future auto update and custom looks downloads will be faster as well.