Deserialize json array into existing Vec

Question

I have some function, here called request, that returns a JSON string with some data. For example, an array with variable size. I can deserialize the JSON string into a Vec just fine with serde_json, but I have to call the request many times. I would like instead to reuse the Vec already created, instead of dropping and creating a new Vec every time deserialization happens.

Is there a way to deserialize a JSON string into an existing Vec? I had a look at the visitor and at this example, but I am not sure which I would have to implement. Either a custom visitor or a deserializer that wraps the serde_json::Deserializer and has a reference to a Vec, or perhaps something else entirely.

Here is some code with an example. Thanks.

use serde_json; fn request(i: usize) -> String { let mut v = vec![0; i]; for n in 1..i { v[n] = n } serde_json::to_string(&v).unwrap() } fn main() { let mut bigvec = serde_json::from_str::<Vec<i32>>(&request(100)).unwrap(); println!("{:?}", &bigvec); // - now we will rerun the request many times and we would like to reuse our vec let many_iterations = 10; for n in 1..many_iterations { bigvec.clear(); let resp = request(n); // we receive a new request with a vec with different length // - we would like something like the below, which does not exist serde_json::from_str_into::<Vec<i32>>(&resp, &mut bigvec); } }

Playground

Locke · Accepted Answer · 2024-06-01 17:15:27Z

There is a hidden¹ deserialize_in_place method on the Deserialize trait that can support this. By default it would simply create a new value and assign it to the existing one, but Vec's impl does exactly what you want.

So you just need to call it manually with the serde-json deserializer:

use serde::Deserialize; use serde_json::Deserializer; let mut deserializer = Deserializer::from_str(&resp); Deserialize::deserialize_in_place(&mut deserializer, &mut bigvec).unwrap();

^{1. This is not a private API. The documentation says "This method is stable and an official public API, but hidden from the documentation because it is almost never what newbies are looking for."}

However, using #[derive(Deserialize)] will NOT implement deserialize_in_place unless you enable the deserialize_in_place feature of serde_derive. The default implementation of Deserialize::deserialize_in_place simply calls to Deserialize::deserialize. This means that calling deserialize_in_place will not always perform in-place deserialization.

#[derive(Deserialize)] struct Foo { a: Vec<i32>, } let mut foo = Foo { a: Vec::new() }; for data in example_data { // If the `deserialize_in_place` feature is not enabled, this call will result // in a new `Vec` being created every call. let mut deserializer = Deserializer::from_str(&resp); Foo::deserialize_in_place(&mut deserializer, &mut foo).unwrap(); }

IMHO making it hidden is a mistake. I've never known about it...
Thanks for the tip. I may have to grep the source code instead of going to docs.rs to find these things. I worked fine for the example above, but as soon as I put the vec in a struct and tried to serialize the struct in place, it did not work anymore. So I think I will have to implement my own version of deserialize in place. Accepting the answer since the more complicated case was not part of my question.

Locke · Accepted Answer · 2024-06-01 16:46:42Z

EDIT: I had forgotten about serde::Deserialize::deserialize_in_place (hidden in documentation) when I wrote this answer. While this answer does show how similar results can be achieved with a Visitor, I recommend following kmdreko's answer instead.

We can't fully deserialize everything in place because serde is designed around returning the value as we deserialize it. However, it you just want to deserialize into a single Vec, we can make a visitor to handle that situation. Here is roughly what that would look like.

use serde::de::{DeserializeOwned, SeqAccess, Visitor}; use serde::{Deserialize, Deserializer}; use std::fmt::{self, Formatter}; struct InPlaceVisitor<'a, T> { buffer: &'a mut Vec<T>, } impl<'de, 'a: 'de, T> Visitor<'de> for InPlaceVisitor<'a, T> where T: Deserialize<'de>, { /// We are deserializing in place so we won't have any output type Value = (); /// Every visitor needs to implement this function to add some /// context to error messages. This will be called if any of the /// `visit_*` functions we didn't implement get called. fn expecting(&self, f: &mut Formatter<'_>) -> fmt::Result { write!(f, "sequence of values") } /// Handle the case where the deserializer finds a sequence of /// values. fn visit_seq<A>(self, mut seq: A) -> Result<Self::Value, A::Error> where A: SeqAccess<'de>, { // Clear the buffer before we begin self.buffer.clear(); // If the deserializer knows the number of items in the // sequence we can ensure we have enough capacity before // starting. However, this is generally only the case for // formats that prefix sequences with their lengths. if let Some(hint) = seq.size_hint() { self.buffer.reserve(hint); } // Now we can just read the values and add them to the // buffer. This will call serde::Deserialize::deserialize for // each element. while let Some(value) = seq.next_element()? { self.buffer.push(value); } Ok(()) } }

Now, one of the tricky parts of using this is that we need to construct our visitor with the buffer we want to push to. This is tricky that means we will not have access to our buffer within serde::Deserialize::deserialize. As a result, we must construct the deserializer ourselves. Not all serde formats expose their Serializer and Deserializer types so this solution won't be possible for all data formats. Luckily serde_json does expose its deserializer, so we can implement a general function to use our visitor.

pub fn deserialize_json_in_place<'de, 'a: 'de, T, R>( reader: R, buffer: &'a mut Vec<T>, ) -> serde_json::Result<()> where T: Deserialize<'de>, R: serde_json::de::Read<'de>, { let mut deserializer = serde_json::de::Deserializer::new(reader); let visitor = InPlaceVisitor { buffer }; // We tell the deserializer what we are hoping to find, then the // deserializer calls the `visit_*` function corresponding to the // type it actually encountered. deserializer.deserialize_seq(visitor) }

With this, we are all set. However for convenience, you may still want to make some simple wrapper functions to handle creating the reader.

pub fn from_str_into<T>(x: &str, buffer: &mut Vec<T>) -> serde_json::Result<()> where T: DeserializeOwned, { let reader = serde_json::de::StrRead::new(x); deserialize_json_in_place(reader, buffer) }

Rust Playground

Thanks for the explanation. Given the other answer above with deserialize_in_place, maybe it is possible to implement deserialize_in_place for a struct using your code suggestion. In my actual use case, I think I will need somehow to merge both of the answers of this post since the default deserialize_in_place impl does not work for a more complicated struct with vecs in it.
@fvall, Oh wow. Ironically, I half remembered something like this existing so I checked the serde documentation and assumed I was mixing it up with another language/library. I never even thought to check for hidden functions. Hopefully this answer is still helpful for showing how you could implement a Visitor. I'll leave this answer up, but I recommend you use kmdreko's solution instead.

Collectives™ on Stack Overflow

Deserialize json array into existing Vec

2 Answers 2

3 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Related