19

I am implementing a function-like procedural macro which takes a single string literal as an argument, but I don't know how to get the value of the string literal.

If I print the variable, it shows a bunch of fields, which includes both the type and the value. They are clearly there, somewhere. How do I get them?

extern crate proc_macro; use proc_macro::{TokenStream,TokenTree}; #[proc_macro] pub fn my_macro(input: TokenStream) -> TokenStream { let input: Vec<TokenTree> = input.into_iter().collect(); let literal = match &input.get(0) { Some(TokenTree::Literal(literal)) => literal, _ => panic!() }; // can't do anything with "literal" // println!("{:?}", literal.lit.symbol); says "unknown field" format!("{:?}", format!("{:?}", literal)).parse().unwrap() } 
#![feature(proc_macro_hygiene)] extern crate macros; fn main() { let value = macros::my_macro!("hahaha"); println!("it is {}", value); // prints "it is Literal { lit: Lit { kind: Str, symbol: "hahaha", suffix: None }, span: Span { lo: BytePos(100), hi: BytePos(108), ctxt: #0 } }" }
Improve this question

5 Answers 5

Reset to default
13

After running into the same problem countless times already, I finally wrote a library to help with this: litrs on crates.io. It compiles faster than syn and lets you inspect your literals.

use std::convert::TryFrom; use litrs::StringLit; use proc_macro::TokenStream; use quote::quote; #[proc_macro] pub fn my_macro(input: TokenStream) -> TokenStream { let input = input.into_iter().collect::<Vec<_>>(); if input.len() != 1 { let msg = format!("expected exactly one input token, got {}", input.len()); return quote! { compile_error!(#msg) }.into(); } let string_lit = match StringLit::try_from(&input[0]) { // Error if the token is not a string literal Err(e) => return e.to_compile_error(), Ok(lit) => lit, }; // `StringLit::value` returns the actual string value represented by the // literal. Quotes are removed and escape sequences replaced with the // corresponding value. let v = string_lit.value(); // TODO: implement your logic here }

See the documentation of litrs for more information.

To obtain more information about a literal, litrs uses the Display impl of Literal to obtain a string representation (as it would be written in source code) and then parses that string. For example, if the string starts with 0x one knows it has to be an integer literal, if it starts with r#" one knows it is a raw string literal. The crate syn does exactly the same.

Of course, it seems a bit wasteful to write and run a second parser given that rustc already parsed the literal. Yes, that's unfortunate and having a better API in proc_literal would be preferable. But right now, I think litrs (or syn if you are using syn anyway) are the best solutions.

(PS: I'm usually not a fan of promoting one's own libraries on Stack Overflow, but I am very familiar with the problem OP is having and I very much think litrs is the best tool for the job right now.)

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

can you expand on what your library does internally to solve this?
@SashaKondrashov I edited my answer and added some explanation. Does this explain it sufficiently?
I was also interested in how it internally works, and your explanation is plenty sufficient to my eyes
13

If you're writing procedural macros, I'd recommend that you look into using the crates syn (for parsing) and quote (for code generation) instead of using proc-macro directly, since those are generally easier to deal with.

In this case, you can use syn::parse_macro_input to parse a token stream into any syntatic element of Rust (such as literals, expressions, functions), and will also take care of error messages in case parsing fails.

You can use LitStr to represent a string literal, if that's exactly what you need. The .value() function will give you a String with the contents of that literal.

You can use quote::quote to generate the output of the macro, and use # to insert the contents of a variable into the generated code.

use proc_macro::TokenStream; use syn::{parse_macro_input, LitStr}; use quote::quote; #[proc_macro] pub fn my_macro(input: TokenStream) -> TokenStream { // macro input must be `LitStr`, which is a string literal. // if not, a relevant error message will be generated. let input = parse_macro_input!(input as LitStr); // get value of the string literal. let str_value = input.value(); // do something with value... let str_value = str_value.to_uppercase(); // generate code, include `str_value` variable (automatically encodes // `String` as a string literal in the generated code) (quote!{ #str_value }).into() }
Improve this answer

4 Comments

I looked into syn, and found it confusing to use. Also it seems overkill for what I was trying to do.
I'm trying to do the same with syn and float literal, and it does not have a .value() and seem impossible to convert into an actual float
@sukmel There's LitFloat::base10_digits to get the number part of the literal (without the suffix, e.g. for 1.23e4f64 it'd give "1.23e4"), and there's LitFloat::base10_parse which you can use to parse it into e.g. a f32 or f64 value.
@Frxsterm I found the former and used it with f64::parse, thanks for the hint about the latter, that's likely the same but it will look better
0

If you want get to the heart of the matter and use syn directly without a macro:

let value = syn::parse::<LitStr>(literal.to_token_stream()).unwrap().value()

or if your token tree is proc_macro2

let value = syn::parse::<LitStr>(literal.to_token_stream().into()).unwrap().value()
Improve this answer

Comments

-1

I always want a string literal, so I found this solution that is good enough. Literal implements ToString, which I can then use with .parse().

#[proc_macro] pub fn my_macro(input: TokenStream) -> TokenStream { let input: Vec<TokenTree> = input.into_iter().collect(); let value = match &input.get(0) { Some(TokenTree::Literal(literal)) => literal.to_string(), _ => panic!() }; let str_value: String = value.parse().unwrap(); // do whatever format!("{:?}", str_value).parse().unwrap() }
Improve this answer

1 Comment

value is already a String, so value.parse() is a noop. Unfortunately, I don't think helps :/
-1

I had similar problem for parsing doc attribute. It is also represented as a TokenStream. This is not exact answer but maybe will guide in a proper direction:

fn from(value: &Vec<Attribute>) -> Vec<String> { let mut lines = Vec::new(); for attr in value { if !attr.path.is_ident("doc") { continue; } if let Ok(Meta::NameValue(nv)) = attr.parse_meta() { if let Lit::Str(lit) = nv.lit { lines.push(lit.value()); } } } lines }
Improve this answer

Comments