Added queryx macro with support for attributes on inferred type

[Lege19]

The syntax for queryx! is as described in #4114.

Does your PR solve an issue?

fixes #4114
fixes #4113

Is this a breaking change?

This is not a breaking change, existing macros should behave exactly the same as before.

TODO

• Tests for queryx!
• Decent errors for incorrect syntax in queryx! (currently bad syntax usually results in macro expansion overflow)
• Agree on syntax

[abonander]

The intention has always been that if you want control over the generated structure, you should be using query_as!(). It's a little more boilerplate but it's going to be much more maintainable in the long run.

This is especially true if you're wanting to implement serde::{Serialize, Deserialize} on it because otherwise you're going to have no idea what the serialized structure is going to look like without having to mentally convert it from the columns listed in the SQL. You'd also have to hand-write the structure anyway if you wanted to add attributes to any of the fields, which you have to in order to get, e.g. time::OffsetDateTime to serialize to a portable format.

If you just wanted to add Debug to the generated type just for, y'know, debugging purposes, I could see adding a configuration option to the new sqlx.toml file, but it needs to be opt-in because some people really care about how much code is generated.

What would be cool is to have some utility that can automatically convert any query!() invocation to a query_as!() invocation, but you can already sort of get that by getting your IDE to show you the result of macro expansion and copy the Record struct from there (shown here with RustRover):

---

I also don't agree with trying to combine the functionality of all the query*!() variants into one macro. The more magical operating modes a single API item has, the harder it is to learn, the harder it is to teach, and the harder it is to keep track of them all. The whole point of the query macros is that it's not an entirely new, bespoke, magic DSL; this seems like a slippery slope down that path. The separate variants also play nicer with IDE support via autocompletion.

I am actually wanting to get rid of some variants, but that's by making the macros more composable with existing mechanisms that the user should already be familiar with (in fact, a lot of people expect this to work already and get frustrated that it doesn't, which is understandable).

---

Having to list column names twice (or more) is annoying, but I want to address that in other ways:

• You should have to use type overrides much less in 0.9.0 because you'll be able to set global overrides instead
• If we implement Replace query_file*!() variants with inherent support for include_str!() and other macros #3388, then you could concat!() your SQL with a macro that returns the list of columns (it'd have to be a macro and not a string const unfortunately)
• If we implement query_as! macros do not use FromRow #514, then you could just use SELECT *
  • Though the server will still send the unused columns, so I'd hesitate to recommend this unless most of the columns are used.
  • The ClickHouse Rust client (which I'm now the maintainer of) has a magic ?fields placeholder which is really convenient; we could potentially adopt something similar here.

[Lege19]

The intention has always been that if you want control over the generated structure, you should be using query_as!(). It's a little more boilerplate but it's going to be much more maintainable in the long run.

I'm not convinced by the claim that it's more maintainable. The more code there is, the more code there is to maintain, and the more duplication there is, the more things there are that need to match.
There's certainly something to be said for making all the types explicit though.

This is especially true if you're wanting to implement serde::{Serialize, Deserialize} on it because otherwise you're going to have no idea what the serialized structure is going to look like without having to mentally convert it from the columns listed in the SQL.

Is it really that much harder to mentally convert from the SQL columns than to mentally convert from the fields of a Rust struct.
Again, the real difference is the lack of explicit types --- but for most serialization formats there are very few types, so it's not like in Rust where it will matter if you get a String or a Box<str>

You'd also have to hand-write the structure anyway if you wanted to add attributes to any of the fields, which you have to in order to get, e.g. time::OffsetDateTime to serialize to a portable format.

True, but this could equally well be solved by adding more features to the macro, which is more practical when it is one macro.

If you just wanted to add Debug to the generated type just for, y'know, debugging purposes, I could see adding a configuration option to the new sqlx.toml file, but it needs to be opt-in because some people really care about how much code is generated

Actually, the code that always derives Debug has been there for 5 years (this commit). I don't know what the behaviors was before that, but the point is that this is not new.

I figured if this was best for the query! it was also best for queryx!, but again, adding more features to this macro --- such as a flag to disable deriving Debug, or even just disabling it by default --- is easier than changing the current family of macros.

What would be cool is to have some utility that can automatically convert any query!() invocation to a query_as!() invocation, but you can already sort of get that by getting your IDE to show you the result of macro expansion and copy the Record struct from there (shown here with RustRover):

I hadn't thought about this and I like this idea, but it doesn't change my opinion on queryx!.

I also don't agree with trying to combine the functionality of all the query*!() variants into one macro. The more magical operating modes a single API item has, the harder it is to learn, the harder it is to teach, and the harder it is to keep track of them all. The whole point of the query macros is that it's not an entirely new, bespoke, magic DSL; this seems like a slippery slope down that path.

The motivation for combining query*!() in one macro is purely about making it more practical to add more features. Considering sqlx is already willing to connect to a remote database server at compile time to check your queries, I have the strong impression that being powerful is higher priority than being simple.
I think the flexibility of these macros is lagging behind what the sqlx backend is capable of accommodating, adding more features to the macros is the obvious way to close this gap.

If there is such resistance to the DSL part though, it might be better to include the same functionality by adding optional keyword arguments by exposing a thinner wrapper over sqlx_macros::expand_query!.

The separate variants also play nicer with IDE support via autocompletion.

I've done some testing on this with Rust Analyzer (I don't have easy access to other setups).

I think the main place you would want and expect autocompletion is in the parameters, and Rust Analyzer seems to understand those as expression position, and stops offering hints if you try and pass too many parameters. This is the same for both.
Rust Analyzer does however provide better autocompletion on the explicit type when using query_as! than queryx!.

#3388, but that's by making the macros more composable with existing mechanisms that the user should already be familiar with (in fact, a lot of people expect this to work already and get frustrated that it doesn't, which is understandable).

I'd be happy to use include_str!(...) instead of file(...). I think just doing this though doesn't really solve the problem with the current macros. You still double the number of macros with each feature you add, or give the macros some syntax, which is what I have tried.

You should have to use type overrides much less in 0.9.0 because you'll be able to set global overrides instead

This is great news which I hadn't been following.

I don't think it completely solves the problem though because plenty of people (including myself) will almost always write out all the columns in their query. So if you want to derive other traits you still need to write it out twice.

If we implement #514, then you could just use SELECT *

I'd love to see this feature, but the old behavior should probably still be available because it generates less code. So... even more macros?

The ClickHouse Rust client (which I'm now the maintainer of) has a magic ?fields placeholder which is really convenient; we could potentially adopt something similar here.

This would be great.

Autocompletion

Since these macros already generate a closure, why not make the closure the output of the macro so that it emits a function that you call with the macro parameters? This seems like it should be possible and then you could even get signature help on the parameters. It should probably call the closure automatically if there are no parameters.

Conclusion

There are clearly multiple features that would be good to add to these macros. It's not practical to have a separate variant for each combination --- I think you agree with me on this.

Therefore there will have to be some macros that accommodate multiple combinations. You could still divide the combinations into multiple macros, but I think you might as well have an everything macro.

Your points about the syntax being harder to manage are completely valid, and I'm open to switching to a keyword based syntax. This is probably also more resilient against needing changes to existing code to accommodate the syntax for new features.

I would say though that there should still be some syntax, even if most features are done using keywords, because it is desirable to still do very simple things with very simple syntax.
For example querying with a string literal, no parameters, and an inferred return type: you should just be able to pass the string literal to the macro and nothing else.

[abonander]

I'm not convinced by the claim that it's more maintainable. The more code there is, the more code there is to maintain, and the more duplication there is, the more things there are that need to match.
There's certainly something to be said for making all the types explicit though.

Except in larger applications you often end up writing multiple queries that return the same structure, anyway. And sometimes you need some post-processing steps to fix up the structure. Having control over the structure's definition is so much more valuable than I think you realize. Ctrl-F ArticleFromQuery in this file for examples: https://github.com/launchbadge/realworld-axum-sqlx/blob/f1b25654773228297e35c292f357d33b7121a101/src/http/articles/mod.rs

I realized that if we implemented #3388, then it'd be very easy to create reusable query fragments for less duplication, e.g.:

macro_rules! select_from_foo( ($tail:literal) => { concat!("SELECT bar, baz, quux FROM foo ", $tail) } ); sqlx::query_as!(Foo, select_from_foo!("WHERE bar = $1"), bar)

We could create a macro to make this even easier:

sqlx::query_fragments! { select_from_foo => "SELECT bar, baz, quux FROM foo", select_from_foo_join_bar => "SELECT bar.*, baz, quux FROM foo INNER JOIN bar USING (bar_id)", } sqlx::query_as!(Foo, select_from_foo_join_bar!("WHERE bar_id = $1"), bar_id)

That doesn't necessarily solve the problem of writing the columns twice, but it saves you having to write them out more than that, which is definitely something.

Is it really that much harder to mentally convert from the SQL columns than to mentally convert from the fields of a Rust struct.

There's actually a double indirection there. To understand what the JSON output should look like, you'd have to understand what the Rust struct should look like, and to understand what the Rust struct should look like, you'd have to parse the SQL to see what columns are being returned. That's a ton of cognitive overhead.

And then there's even more temptation to be really lazy and just do SELECT *, and potentially leak a bunch of confidential or proprietary information by accident. And you'd have zero idea that there's a bug just from looking at the code because you'd also have to intimately understand the database schema.

Experience tells me this feature would just be a massive footgun, and I'm really not comfortable with that.

Actually, the code that always derives Debug has been there for 5 years (this commit). I don't know what the behaviors was before that, but the point is that this is not new.

I honestly forgot we had that, and I didn't notice it in the screenshot because it ended up on the previous line for some reason. Maybe an artifact of how we concatenate token streams?

The motivation for combining query*!() in one macro is purely about making it more practical to add more features.

One big problem is that it's much, much less practical to document, and to teach. The docs for query!() itself are already super long and difficult to navigate and link into (it should probably be pared down to just reference material, and most of the prose moved to a separate guide doc, but that's an orthogonal concern); if we merged the docs for all the operating modes together, with examples (because some people really do need examples to understand how it works), it'd easily be double or triple the length.

You didn't really run into this because you just link to the docs for all the other macros: https://github.com/launchbadge/sqlx/pull/4115/files#diff-5ee96057bd109a82372c87c64ed6c0b59a16257e72311971b642523a3c68dd6fR875-R885

The separate macros provide a very natural, easily navigable boundary between doc sections, and it plays much better with how rustdoc works, especially searching.

Considering sqlx is already willing to connect to a remote database server at compile time to check your queries, I have the strong impression that being powerful is higher priority than being simple.

query!() itself is designed to be simple. It's a convenience wrapper for when you just want to get some data in/out. The fact that its effects are inherently constrained to a single function is a feature, not a bug.

The whole point has been that if you need more control than that, you should really be using query_as!().

This is not all that different from how Rust allows the types of local variables to be inferred but requires function signatures to be fully specified. While yes, a significant motivation was implementation complexity, it also turns out that having abstractions with well defined interfaces is extremely important for maintainable software.

I think the flexibility of these macros is lagging behind what the sqlx backend is capable of accommodating, adding more features to the macros is the obvious way to close this gap.

I have no intention of adding more variants. Like I said, I'm trying to get rid of some of them. As you've seen, 90% of the complexity isn't in having multiple macros anyways.

#514 wouldn't be a new operating mode; it'd be a breaking change to query_as!() itself. I suppose we could use autoref specialization to introduce support backwards-compatibly, but I think that'd be too magical.

And at the end of the day, "too magical" is really the issue here. I don't want these macros to become a magical black box where you put SQL in and get JSON out, and the fact that it's mapped to a Rust struct is just an implementation detail. I've really come to despise systems like that, because when things go wrong you have very little capability to introspect the system and try to understand what's gone wrong.

It also makes it a nightmare to find your way around a new project. You have to learn how the black box works to have any idea what's going on. The further a black box's effects can reach, the more lost you are until you figure it out.

With query!(), the context is entirely localized. You can see the field accesses on the struct and how they correspond to the query, and infer the correlation with relative ease. With query_as!(), you just have to look at the struct itself and don't really have to worry about how it gets populated if that's not the part you're currently concerned with.

With this, you kind of have... the worst of both worlds.