Timeline for How can I speed up slow loading of big levels?
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 6, 2017 at 18:48 | comment | added | TemporalWolf | We've seen almost 100x improvements in speed simply by switching parsers (from Python to C) which was extremely low hanging fruit (import a different module, change 3 lines of code.) In this benchmark, XML was almost twice (+84%) as verbose as JSON when storing the same data. Reducing it to binary would yield even better results. It takes me about that much time to load & parse files which are 4GB in size on disk (~21GB once loaded into python in RAM)... | |
| Dec 6, 2017 at 13:59 | comment | added | Damon | If you can live with an occasional small "hiccup" then you need not even worry about touching pages. Just map and forget. | |
| Dec 6, 2017 at 13:58 | comment | added | Damon | @Thierry: Indeed. The nice thing with a binary (non-text) format where you have explicit lengths of everything is that you can just skip ahead a gigabyte worth of data if you know from the length that the stuff you want is there. XML cannot provide that. Assuming large enough address space (we're not quite in 32-bit-is-history land yet, but hopefully soon) you can even just map one single GB-sized file, and let the VM manager do the work. Touching pages a few frames before actually accessing them (worker thread) so the main thread never sees a fault, this works better than one would hope. | |
| Dec 6, 2017 at 13:34 | comment | added | Thierry | While this answer is correct, I think it's important to stress that this only gives a constant factor improvement. This is probably sufficient for appmaker1358's use case; as it would bring the load time down from 45s to something manageable. If the load time of the level map would be 450s, though, (e.g. in an Open World setting) this approach would not suffice, and the original idea of splitting up into chunks would be needed. Note that these ideas are complementary. The different chunks could even be part of the same file if it's in a parse-less binary format with explicit item lengths. | |
| Dec 6, 2017 at 9:50 | history | answered | Damon | CC BY-SA 3.0 |