7
\$\begingroup\$


Im writing a serializer that will do serialization/deserialization as fast as possible, and that uses templates so that i dont have to create a ton of functions for every data structure,

This is the part of it that pre-calculates the size of the data to be serialized

I have only tested this on MSVC and GCC

So this code has get_size_needed() which should get the total size (in bytes) needed to store the data for the arguments.

You should be able to put any container/arithmetic type (and any combination of those types) into it as a argument and it should calculate its size at compile time.

It should also support calculating the size of nested containers.

sorry if its messy, this is my first time dealing with templates and concepts (with some help from some people)

Serialization.h

#include <vector> #include <cstdint> #include <string> #include <ranges> #include <numeric> #include <type_traits> #ifdef _MSC_VER // vvv MSVC implementation vvv (MSVC does not support <ranges> nor templates with auto arguments) // checks the requires size of a variable template <typename T> constexpr std::size_t get_size_needed_for_type(const T& value) { // If the value is a bool/int/float then just get the size of it if constexpr (std::is_arithmetic_v<T>) { return sizeof(value); } // If the value is a container, then get the size of the container, and the size of its elements else if constexpr (std::is_class_v<T> || std::is_union_v<T>) { // Get the size of every element return std::accumulate(value.cbegin(), value.cend(), std::size_t{ 0 }, [](std::size_t acc, const auto& v) { return acc + get_size_needed_for_type(v); }); } // I dont know what whould end up here, but its probably not something thats serializable else { static_assert(false, "Unsupported type for serialization!"); } } #else // vvv GCC implementation vvv - ^^^ MSVC implementation ^^^ // Only accepts bool/int/float template <typename T> requires std::integral<T> || std::floating_point<T> constexpr std::size_t get_size_needed_for_type(const T& value) { return sizeof(value); // return the size of it } // If the is a range, then its probably a container, then get the size of the container, and the size of its elements constexpr std::size_t get_size_needed_for_type(const std::ranges::range auto& value) { // Get the size of every element return std::accumulate(value.cbegin(), value.cend(), std::size_t{ 0 }, [](std::size_t acc, const auto& v) { return acc + get_size_needed_for_type(v); }); } #endif // ^^^ GCC implementation ^^^ // Iterate through arguments, and accumulate the size of all of them template <typename... Args> constexpr std::size_t get_size_needed(const Args&... args) { return (... + get_size_needed_for_type(args)); }

Whould be nice to get thoughts / criticism / optimization tips on this

\$\endgroup\$

2 Answers 2

Reset to default
4
\$\begingroup\$

Stick with supporting the lowest common denominator

I don't see a difference in functionality in the GCC and MSVC version of your code, so I wouldn't bother writing two implementations. That is just repeating yourself in a way.

Also note that it's not really GCC vs MSVC, it is some version of GCC and some version of MSVC that you checked, that doesn't mean all versions of those compilers have the same difference. Older versions of GCC might also not support some C++20 features that you used, and newer versions of MSVC will likely support those at some point. Also, what about Clang, ICC and all the other compilers out there?

Use feature testing macros

If possible, use feature testing macros instead of testing for compiler brand to check whether you can use a given feature.

You can't serialize all containers by just looking at their contents

Containers can be more than just their contents. The ordered STL containers for example allow you to provide them with a functor that tells them how the elements are to be ordered. And if you go outside the STL, consider what happens if you have for example a class that implements a binary tree, and that provides begin() and end() functions to traverse the tree in-order. If you serialize just the elements, you miss the way the nodes in the tree are linked together, which might be significant information.

\$\endgroup\$
2
  • \$\begingroup\$ I implemented this not thinking to add support for most containers, but rather go lean and mean to only support the most common containers. (Whould be nice to restrict it to those, but i have no idea how). The "MSVC vs GCC" thing is more me implementing a temporary replacement for MSVC, since it didnt support ranges and concepts. (and then remove the replacement once MSVC gets those features) \$\endgroup\$ Commented Oct 21, 2020 at 21:59
  • \$\begingroup\$ This is meant for internal use in my application rather then as a library. so this whould replace hundreds of functions that did serialization slightly differently \$\endgroup\$ Commented Oct 21, 2020 at 22:08
3
\$\begingroup\$

In general, you cannot implement fully automated serialisation without Reflection which is currently unavailable in C++. You can only specify rules for variety of templates like std::vector<T> or std::set<T> so it will be automated for them as well as you can apply memcpy for trivially copyable types. But for other types you'll need to write the serialisation function yourself.

I don't understand why you need to precalculate the size. Just keep a fixed sized buffer and keep filling it until its full and then forward it to the file or whatever you want to use to store it. Why bother precalculating the total size? You increase upkeep cost by requiring to write extra functions just to serialise/deserialize the data and in addition the calculating the size. Think how much time it will take to compute it for std::vector<char> of huge size - at least in the implementation you wrote.

Besides, just the fact that the object is a range doesn't mean that it should be serialised by storing its elements one by one. It's just a really bad and naive approach to the task. How do you plan to figure out the number of elements when you deserialize it? You could make a general solution for ranges but it will require you to make extra type trait that indicates that its data should be stored by saving element by element.

\$\endgroup\$
4
  • \$\begingroup\$ In my testing the constexpr functions has made the compiler do all the work so that it didnt have to check sizes at compile time (even with resizable vectors) The precalculating of size is because i dont want to have huge buffers allocated, so id rather calculate it right before i put the data there \$\endgroup\$ Commented Oct 21, 2020 at 22:05
  • 1
    \$\begingroup\$ @OptoCloud you messed up with the testing... that's not how constexpr works. At most compiler optimization could optimize out the dynamic range case. But once any live code runs the calculation have to be run at runtime. \$\endgroup\$ Commented Oct 22, 2020 at 0:29
  • \$\begingroup\$ oh, yea i know that, but the compiler adds all the static sized variables together, and makes it so at runtime it just adds that with sizeof(type) * container.size() \$\endgroup\$ Commented Oct 22, 2020 at 12:17
  • 1
    \$\begingroup\$ @OptoCloud yeah, but it will increment it one by one for the whole containers size instead of just adding a single sizeof(type)*container.size() unless optimizer figures it out. At any rate it is unreliable and will screw performance of debug mode. And worst of all, the computation serves no purpose. \$\endgroup\$ Commented Oct 22, 2020 at 12:44

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.