12
$\begingroup$

Working in the creations of APIs with Mathematica 12.2, I got this problem with diacritics when sending JSON data.

Here is a toy code using httpbin.

HTTPRequest["https://httpbin.org/anything", <| Method->"POST" ,"Body" -> (<|"textWithDiacritics"-> "àéíóú"|>//ExportString[#, "RawJSON"]&) |> ]//URLRead[#, "BodyBytes"]&//FromCharacterCode//ImportString[#, "RawJSON"]&

enter image description here

I found this related post, It seams to be a bug in ExportString for RawJSON.

Any suggestions on how to get around this case?

Cross post in Wolfram Community

Improve this question
$\endgroup$

3 Answers 3

Reset to default
9
$\begingroup$

ExportString JSON produces UTF8 encoded json string, if put as HTTPRequest body it will undergo another encoding unless you prevent it with an option:

HTTPRequest[..., CharacterEncoding -> None]

It is analogous to the double decoding issue which you address with bodybytes//FromCharacterCode//ImportString.

Exchanging JSON via http requests

This is very common these days so for the record:

Building requests (for URLRead input):

HTTPRequest[ url_ , <| "ContentType" -> "application/json" , "Body" -> ExportString[ jsonCompliantExpr, "RawJSON"] , "Method" -> "POST" , "Headers" -> {"Accept"->"application/json"} (* less important *) |> , CharacterEncoding -> None ]

Receiving requests ( for APIFunction )

I will skip the part of handling different payloads or error handling, just assuming the incomming request is on generated as above will need:

APIFunction[ {} , Function[whatever (*we don't care about query params*) , HTTPRequestData["BodyBytes"] // FromCharacterCode // ImportString[#, "RawJSON"]& // doSomethingWithAssoAndGenerateResponse ] ]

Building responses (* in API/FormFunctions *)

For APIFunction etc there is a nice wrapper ExportForm which works with GenerateHTTPResponse which is called on the result of APIFunction's function.

So a quick way:

 APIFunction[{}, ExportForm[association, "RawJSON" ]& ]

And a longer way:

HTTPResponse[ ExportString[association_ , "RawJSON" ] , <| "ContentType"->"application/json" |> , CharacterEncoding -> None ]

I wish there was ExportForm support for building HTTPRequests, I even asked WRI, the status is 'maybe'.

Parsing responses (* what URLRead needs to do *)

Worth to mention that URLExecute does it well wrt encoding but I always need more controll on status codes etc so I need to use URLRead. So as you mentioned:

URLRead[...]["BodyBytes"]& // FromCharacterCode//ImportString[#, "RawJSON"]&

I would not try to decode "Body" because it is very unclear what the body is according to developers, the behavior of "Body" worsened in 12.2 it became even less predictable. So URLExecute or "BodyBytes" are the way to go.

More reading:

HTTPRequest syntax problem

Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is cool too. Now I don't know which one to accept! :) $\endgroup$ Commented Mar 11, 2021 at 19:36
  • $\begingroup$ @Murta thanks for the accept. I added a guideline section so you won't be sorry :) $\endgroup$ Commented Mar 12, 2021 at 5:58
8
$\begingroup$

I've had luck using the functions WriteRawJSONString and ReadRawJSONString from Developer context. Check out: WriteRawJSONString vs ExportString?

HTTPRequest[ "https://httpbin.org/anything", <|Method -> "POST", "Body" -> (<|"textWithDiacritics" -> "àéíóú"|> // Developer`WriteRawJSONString)|>] // URLRead[#, "BodyBytes"] & // FromCharacterCode // Developer`ReadRawJSONString (* <|"args" -> <||>, "data" -> "{ \"textWithDiacritics\":\"àéíóú\" }", "files" -> <||>, "form" -> <||>, "headers" -> <|"Accept" -> "*/*", "Accept-Encoding" -> "deflate, gzip", "Content-Length" -> "38", "Content-Type" -> "text/plain;charset=utf-8", "Host" -> "httpbin.org", "User-Agent" -> "Wolfram HTTPClient 12.2", "X-Amzn-Trace-Id" -> "Root=1-604a65d2-560dfd4155a830d409b373c9"|>, "json" -> <|"textWithDiacritics" -> "àéíóú"|>, "method" -> "POST", "origin" -> "165.254.183.48", "url" -> "https://httpbin.org/anything"|> *)
Improve this answer
$\endgroup$
1
  • $\begingroup$ This is great, tks Chuy. $\endgroup$ Commented Mar 11, 2021 at 19:35
5
$\begingroup$

For what it's worth, the issue is in how ExportString is handling the diacritics. We can see that it botches them, but the function being used under the hood doesn't

(<|"textWithDiacritics" -> "àéíóú"|> // ExportString[#, "RawJSON"] &) "{ \"textWithDiacritics\":\"àéíÃ\.b3ú\" }" <|"textWithDiacritics" -> "àéíóú"|> // Developer`WriteRawJSONString "{ \"textWithDiacritics\":\"àéíóú\" }"

That issue can be traced back to the way ExportString works (I think): it opens an OutputStream, uses a standard exporter to write there, then reads back into Mathematica as a "String", because it's fast.

We can see that issue clearly if we mimic that approach:

meh = OpenWrite[]; Developer`WriteRawJSONStream[meh, <|"textWithDiacritics" -> "àéíóú"|>]; Import[Close[meh], "String"] "{ \"textWithDiacritics\":\"àéíÃ\.b3ú\" }" meh = OpenWrite[]; Developer`WriteRawJSONStream[meh, <|"textWithDiacritics" -> "àéíóú"|>]; Import[Close[meh], "Text"] "{ \"textWithDiacritics\":\"àéíóú\" }"

So what's the solution? Well, I think you can just leave the body as an expression and Mathematica will write it out as a raw JSON string for you, i.e. this works for me

HTTPRequest[ "https://httpbin.org/anything", <|Method -> "POST", "Body" -> <|"textWithDiacritics" -> "àéíóú"|> |> ] // URLRead[#, "BodyBytes"] & // FromCharacterCode // ImportString[#, "RawJSON"]["form"] & <|"textWithDiacritics" -> "àéíóú"|>

But in any case, any of the existing solutions provide real answers, this was mostly just to point how what's going wrong.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.