For example, how would a macro that defines custom control structures differ from just writing a higher-order function?
For the most part, anything you can do with macros can be done with higher-order functions. Macros just allow you to do it with simpler syntax.
This allows you to write
(my-when (>= x y) (format *error-output* "~&~d must be less than ~d" x y))
instead of the more verbose
(my-when (lambda () (>= x y)) (lambda () (format *error-output* "~&~d must be less than ~d" x y)))
Without macros, code would be littered with all these lambdas, making it harder to read. Functional programming generally languages solve this littering problem by providing more terse syntax for lambdas.
[Personal anecdote: When I was first learning to program, I ran into the source code to ELIZA, a very simple chatbot that was written in the 70's. About all I remember was that it seemed like every other line began with LAMBDA, which seemed like Greek to me (the only programming language I knew at the time was BASIC). Now I know that the LET macro later obviated all these lambdas.]
In fact, it's not uncommon for macro writers to code them as transforming into calls to higher-order functions, e.g.
(defmacro my-when (condition &body body) `(my-when-func (lambda () ,condition) (lambda () ,@body))) (defun my-when-func (condition-func body-func) (if (funcall condition-func) (funcall body-func)))
This style makes it easy to avoid the "hygiene" problems that are often encountered if the macro needs to introduce local variables in its expansion. I encountered this style when reading the Lisp Machine source code -- macros like WITH-OPEN-FILE and CONDITION-CASE were all written this way.
I expected this to behave like a regular function, but it seems macros are expanded before runtime, and I’m not entirely sure how that affects variable scope and evaluation.
That's correct. When a macro invocation is encountered, it is expanded and then the result is evaluated in place of the original expression. If it expands into another macro invocation, the process repeats until you get a non-macro expression.
As far as variable scope is concerned, it's as if you'd written the expanded code in the first place. You can use MACROEXPAND to see what the expansion of a particular macro invocation would be.
This is not unlike how macros are processed in languages like C, except that it's not just simple string substitution, you have the full power of the Lisp language available to produce the replacement. For example, you can write loops in the expansion function; this is common in macros that have syntax similar to LET, so you can process the variable-length list of bindings.
This is what allows you to write macros like LOOP, which implements a language of its own. The implementation of this contains a simple parser.
Macros are also expanded by the compiler, so there's no run-time overhead.
