dev

# Add macros and the rest of a professional RISC-V assembler

The RISC-V front end already encodes instructions and a handful of data

directives. It does not assemble the files you actually write. Those are GNU as:

`.macro` / `.endm`, `\name`, numbered local labels, `.option`, `.pushsection`.

`r-test/fp.S` is the concrete target.

This document tells you how to add that, in order. Do not expand macros inside

the ANTLR grammar. The encoder already advances `address` in parser actions; a

`.macro` body that is a `lines` rule will emit bytes at definition time, or

fight the address, or both.

## The one-line version

Parse once to collect directives, expand macros / conditionals / includes into

plain source, then parse again to encode. New language features that invent

text belong in a preprocessor. New language features that invent bytes

(`.align`, `.globl`, `.option`) belong in the grammar and `RISCVEncoder`.

```

source .S  →  preprocessor  →  ANTLR parse + encode  →  bin / ELF

```

Dialect for everything new is GNU as. Keep the NASM `%define` / `%include` that

already work so `testbench/define.s` does not break. Do not add a second full

NASM `%macro` unless a later day needs it.

## What already exists

Do not reinvent these. They are incomplete, not absent.

| Piece | Where | What it actually does |

|---|---|---|

| `preProcess` | `AssemblerLib.java` | Parse, then string-replace `%include` and `%define` |

| `%define` | lexer `DEFINE`, `DefineListener` | Flat name → value. No parameters. Redefine calls `System.exit` |

| `%include` | lexer `INCLUDE`, `IncludeListener` | Inlines a file. Search path is hardcoded `testbench/` |

| `%ifdef` / `%elif` / `%else` / `%endif` | grammar `ifdef` | Parsed. Never evaluated |

| `%times` / `times` | lexer `TIMES`, `preProcess` | Empty `if (content.contains("%times"))` |

| `.byte` `.half` `.word` `.dword` `.string` | grammar `dot*` | Encoded. `.string` takes `IDENTIFIER`, not a quoted string |

| `. IDENTIFIER` | grammar `section` | Stores a name for the listing. No real section |

| labels | grammar `label` | `IDENTIFIER COLON` only. No `1:` / `1b` / `1f` |

| comments | lexer `LINE_COMMENT` | `;` only. GNU as uses `#` |

| listing | `-l`, `RISCVEncoder.listing` | Records the line the parser saw, not an expansion |

| ELF | `Assembler.java` `-f elf` | Already writes via `executablelibrary` |

`DefineListener.map` and `DefineListener.lines` are static. A second file in the

same JVM inherits the first file's defines. Make them instance state when you

touch that class.

The grammar rule named `macro` is not a macro. It is a bucket for define,

ifdef, include, and data:

```

macro : define | ifdef | include | dotbyte | dothalf | dotword | dotdword | dotstring ;

```

Leave that name alone until the preprocessor owns define/ifdef/include. Then

the rule can shrink to data directives.

## Why the preprocessor, not the grammar

`RISCVAssemblerParser` members hold `address` and every instruction does

`address+=encoder.encodeType...(...)`. A `.macro ADD rd, rs` whose body is

parsed as `lines` will encode `add` while you are still defining `ADD`. The

invocation then has nothing to expand, or encodes a second time.

`preProcess` already does the right shape: walk the text, produce new text,

then the real `assemble()` encodes. Grow that into a package

`hk.quantr.assembler.riscv.preprocess` rather than piling more static maps

into `AssemblerLib`.

`Assembler.main` must call the preprocessor before the encode parse. Today

`preProcess` is used from tests (`TestMacro`, `TestMissingMacro`) and is easy

to skip from the CLI path. Grep `preProcess(` and make every assemble entry

go through it.

## Files you will touch

```

src/main/java/hk/quantr/assembler/antlr/RISCVAssemblerLexer.g4

src/main/java/hk/quantr/assembler/antlr/RISCVAssemblerParser.g4

src/main/java/hk/quantr/assembler/AssemblerLib.java

src/main/java/hk/quantr/assembler/Assembler.java

src/main/java/hk/quantr/assembler/riscv/listener/DefineListener.java

src/main/java/hk/quantr/assembler/riscv/listener/IncludeListener.java

src/main/java/hk/quantr/assembler/riscv/RISCVEncoder.java

src/main/java/hk/quantr/assembler/riscv/preprocess/   (new)

src/test/java/hk/quantr/assembler/riscv/TestGasMacro.java  (new)

```

After any `.g4` change: `mvn -DskipTests compile` so ANTLR regenerates into

`target/generated-sources/antlr4`.

## Phase 1: real macros

This is the feature the request named. Do it first and stop to test.

### Tokens

In `RISCVAssemblerLexer.g4`, next to `DEFINE` / `INCLUDE`:

```

DOTMACRO    :   '.macro';

DOTENDM     :   '.endm';

DOTEXITM    :   '.exitm';

```

Do **not** add a parser rule that treats the body as `lines`. The preprocessor

reads these as lines of text.

### Data

A definition is a name, a list of formal parameters, and the raw body (lines

between `.macro` and `.endm`, not encoded). An invocation is a name used as an

opcode with comma-separated arguments.

GNU as substitution:

| In the body | Becomes |

|---|---|

| `\formal` | the matching argument |

| `\\@` | a counter that increments every invocation (unique local labels) |

| `\()` | concatenator, so `\a\()b` is arg `a` then the letter `b` |

GAS also allows `.macro NAME arg1=default`. Defaults can wait until the

required-argument form works.

### Algorithm

1. Walk the source line by line. `#` and `;` comments strip for this walk, but

   keep the original line text for error reporting.

2. On `.macro NAME [formals...]`, slurp until `.endm`. Nested `.macro` inside

   a body is stored, not executed. Missing `.endm` is an error that names the

   opening line.

3. On a line whose first identifier is a defined macro, split arguments on

   commas that are not inside `(...)` or quotes, bind formals, substitute,

   splice the expansion in place of the invocation, and re-scan from there so

   macros can call macros.

4. Recursion depth: cap at something like 100. The error must name the

   invocation site and the definition site.

5. Unknown arity: error, do not encode a truncated body.

Invocations look like instructions (`SECTION name`, `T fld`, `LDD fs0, d_one`).

The encode parse must not see those names. After expansion, `SECTION` is gone

and the body (`la`, `call`, ...) remains.

### What not to put in the grammar

Do not add `NAME args` as a generic instruction alternative. That would steal

real opcodes (`add`, `ld`) if someone names a macro after one. Expansion

happens first; the parser only ever sees real instructions.

### First golden test

`r-test/fp.S` macros, smallest useful subset:

```asm

	.macro	E

	la	a3, 7b

	call	test_end

	.endm

	.macro	LDD freg, sym

	la	a0, \sym

	c.fld	\freg, 0(a0)

	.endm

```

A file that does `LDD fs0, scratch` then `E` must expand to `la` / `c.fld` /

`la` / `call`. Compare bytes to `riscv64-elf-as` (phase 1 can ignore

`.option` and `.pushsection` by not using those lines yet).

`TestGasMacro` outline:

```java

String src = Files.readString(Path.of("src/test/resources/macro_ldd.s"));

String expanded = Preprocessor.expand(src, "rv64");

byte[] ours = assembleRv64(expanded);

byte[] gas = gasAssemble(src);   // riscv64-elf-as -march=rv64imafdc

assertArrayEquals(gas, ours);

```

`gasAssemble` writes a temp `.s`, runs `riscv64-elf-as -o t.o`, then

`riscv64-elf-objcopy -O binary t.o t.bin` (or read `.text` from the ELF).

Same idea as `FullTest` versus gas.

Phase 1 is done when that test passes and `testbench/define.s` still works.

## Phase 2: conditionals and repeats

`%ifdef` is already in the grammar and does nothing. Evaluating it in the

parser would still encode the false branch, because both `lines` alternatives

are walked. Evaluate in the preprocessor, then delete the `ifdef` parser rule

or leave it as a no-op that never fires on expanded text.

Add GAS forms. These are what `fp.S` does not use yet but every real tree has:

```

.if  expr

.ifdef name

.ifndef name

.else

.endif

.rept N

.endr

```

`.if` uses the same expression evaluator as immediates

(`CalculatorLibrary.cal`). A name in `.ifdef` is defined if it is in the

`%define` / `.equ` table or is a `.macro`.

`.rept N` splices the body `N` times, then re-scans. That is also the correct

implementation of `times` / `%times` (the empty branch in `preProcess`).

`.irp reg, t0, t1, t2` can wait until `.rept` works. It is the same loop with

a formal rebound each iteration.

False branches must not define macros and must not invoke them. Skip lines

until the matching `.else` / `.endif`, tracking nest depth. A dangling

`.endif` is an error.

## Phase 3: symbols and local labels

Without this, expanded `fp.S` still will not assemble.

### `.equ` / `.set`

Same table as `%define`. `.set` may redefine; `%define` today forbids it and

exits. Pick one policy and document it: `.set` overwrites, `%define` of an

existing name is an error (keep today's behaviour).

Lexer: `DOTEQU : '.equ'; DOTSET : '.set';`

The encode parse needs these names in immediates. Either the preprocessor

substitutes them (like `%define` already does with `replaceAll("\\b"+name+"\\b")`)

or `CalculatorLibrary` consults the table. Substitution is simpler and matches

the current `%define` path.

### Numbered local labels

GNU as: `1:` through `19:` (you only need `0`–`9` to start), referenced as

`1b` (nearest backward) and `1f` (nearest forward). `fp.S` uses `7:` / `7b`

and `9:` / `9b`.

This cannot be only a preprocessor rewrite of the current file, because `1f`

depends on a label that appears later. Do it in the encoder:

1. Lexer: allow `label` to be `[0-9]+ COLON`, and immediates / jump targets to

   be `[0-9]+ [bf]`.

2. First pass, or a collected list: each `N:` at address A is pushed on a

   per-digit list.

3. When encoding `jal` / `beq` / `la` that uses `7b`, take the last `7:` whose

   address is `<=` current address; `7f` takes the next one after.

If you stay strictly one-pass, `7f` is unknown when you see it. The encoder

already has `labels` on the parser (`ArrayList<Label>`). Either two-pass

(walk once for labels, once to encode) or record a fixup and patch the

immediate when the forward label is defined. Two-pass is less clever and

matches how gas works. The current `address+=` one-pass is why forward

regular labels are already shaky; this is the moment to make a label pass

explicit if you have to fight it.

### `#` comments

```

LINE_COMMENT : [;#] ~[\r\n]* ;

```

GAS also treats `/* */` as comments. Not required for `fp.S`. `#` is.

## Phase 4: sections and options

Needed for a real object, and for `fp.S` after macros expand.

### Sections

Replace the catch-all

```

section : DOT sectionName=IDENTIFIER IDENTIFIER? ;

```

with explicit directives so `.macro` / `.equ` / `.option` are not eaten as

section names (today `.` plus an identifier is a section).

```

.text

.data

.rodata

.bss

.align  N

.globl  name

.asciz  "string"

.ascii  "string"

.pushsection name

.popsection

```

`.byte` already exists. Add `.asciz` (NUL-terminated) distinct from `.string`

if `.string` stays identifier-only; or teach `.string` to take

`DOUBLE_QUOTATION ... DOUBLE_QUOTATION`. `fp.S` uses `.asciz`.

`RISCVEncoder` must keep a current section and a stack for

`.pushsection` / `.popsection`. Bytes go into that section's buffer, not one

flat `out`. ELF output (`-f elf`) already exists; point it at those buffers

instead of a single blob. `bin` output can concatenate `.text` then `.data`

the way a raw image expects, or refuse and require `-f elf`. Pick one and

test it.

`.align N` on RISC-V gas is power-of-two (`.align 3` means 8 bytes). Pad with

zeros or `nop`/`c.nop` in `.text`. Wrong interpretation here will fail every

gas comparison.

### `.option`

```

.option rvc

.option norvc

.option push

.option pop

```

`LDD` in `fp.S` wraps `c.fld` in `.option push` / `rvc` / `pop`. Without this,

either you always accept compressed (current behaviour) or you reject `c.fld`

when someone writes `.option norvc`. Store a stack of flags on the encoder.

`encodeType*` for compressed opcodes checks `rvcEnabled`.

## Phase 5: professional tooling

Do this after the language works. None of it changes what bytes mean.

- **Include path.** `preProcessIncludeFile` opens `"testbench/" + to`. Resolve

  against the including file's directory, then each `-I` from `Assembler.main`.

  Add the CLI option next to `-a` / `-o`.

- **Listing of expansions.** `-l` already writes `listing`. After preprocess,

  each encoded line should carry the invocation that produced it, so a failure

  in an expanded `T fld` names `T` and the `.macro T` line.

- **Errors.** Invocation site plus definition site, for macros, includes, and

  missing `.endm`. Stop calling `System.exit` from `DefineListener` /

  `preProcess`; throw or record on `MessageHandler` and let `main` set the

  exit code. Tests cannot survive an exit.

- **Instance state.** `DefineListener.map` and friends must not be static.

## How to test, every phase

Always compare to GNU as. That is the project's existing contract

(`FullTest` / gas vs quantr).

```sh

# gas

riscv64-elf-as -march=rv64imafdc -o gas.o t.s

riscv64-elf-objcopy -O binary --only-section=.text gas.o gas.bin

# ours, after you wire preProcess into main

java -jar target/assembler-*-jar-with-dependencies.jar -a rv64 -f bin -o ours.bin t.s

cmp gas.bin ours.bin

```

Keep a table of fixtures under `src/test/resources/riscv/`:

| File | Phase | What it proves |

|---|---|---|

| `define_still_works.s` | 1 | `%define` + `addi` unchanged |

| `macro_ldd.s` | 1 | `.macro` args, `\formal` |

| `macro_nested.s` | 1 | macro calls macro |

| `macro_count.s` | 1 | `\\@` unique labels |

| `ifdef_false.s` | 2 | false branch emits nothing |

| `rept_3.s` | 2 | body appears three times |

| `local_label.s` | 3 | `1: ... j 1b` |

| `option_rvc.s` | 4 | `c.fld` only with `.option rvc` |

| `pushsection.s` | 4 | string lands in `.rodata`, code in `.text` |

`testbench/define.s` and `testbench/macro.s` stay green.

## Acceptance

Phase 1–4 together: this jar assembles `r-test/fp.S` (or a copy with the

`#` comments and `.equ` values it already has) and the `.text` bytes match

`riscv64-elf-as -march=rv64imafdc`. That file uses every phase: macros with

arguments, `#` comments, `7:` / `7b`, `.option push/rvc/pop`, `.pushsection`,

`.asciz`.

Until that works, do not start ia32 macros, a C preprocessor, or a relocating

linker. Those are different jobs.

## Out of scope

- Rewriting the encoder into a full relocating linker (relocs, `-shared`)

- ia32 / NASM `%macro` / `%endmacro`

- Running `cpp` for `#include` / `#ifdef` / `#define` (GAS `#` is a comment,

  not cpp, unless you pass `-x assembler-with-cpp`)

- Changing instruction encodings or the disassembler