Mandelbrot Renderer in x86-64 Assembly (NASM)

Key Concepts:

• x86-64 Assembly (NASM) programming
• System V AMD64 ABI compliance
• C standard library functions
• Double-precision floating-point arithmetic
• Algorithm implementation from the Mandelbrot set mathematical definition

Argument Parsing and File Output

To begin, my goal was to simply accept command-line arguments and generate a valid image file.

I wanted my program to take three command-line arguments: ./mandelbrot image_size iterations output_file

• image_size: determines both the width and height of the image
• iterations: the number of times to apply the Mandelbrot function per pixel
• output_file: the path where the generated image will be written

I used functions from the C standard library to handle parsing and file I/O:

atoi to convert command-line strings into integers:

…
; size = atoi(argv[1])
mov rdi, [rbx + 8]    ; argv[1]
call atoi    ; convert image size ascii to int in eax
mov r12d, eax    ; IMAGE SIZE = r12d (using rXd registers for lower 32 bits)
…
; iterations = atoi(argv[2])
mov rdi, [rbx + 16]    ; argv[2]
call atoi    ; iterations ascii to int in eax
…

fopen to open the output file in binary mode:

…
; FILE* fp = fopen(path, "wb")
mov rdi, r14    ; filename
mov rsi, write_mode    ; mode: write binary
call fopen    ; FILE* = rax
…

The image is written using the binary PGM format, which has this structure:

P5
width height
255
raw pixel bytes

fprintf to write the PGM header:

…
mov r15, rax    ; FILE* = r15

; fprintf(fp, header_fmt, size, size)
mov rdi, r15    ; FILE*
mov rsi, header_fmt    ; format
mov edx, r12d    ; first %d width
mov ecx, r12d    ; second %d height
mov eax, 0    ; for variadic functions, I'm passing no vector registers
call fprintf
…

To generate the image, I used nested loops over rows and columns, assigning every pixel a constant grayscale value of 128.

fputc to write pixel data:

…
; fputc(128, fp) (take int and write as byte)
  mov edi, 128    ; character (int)
  mov rsi, r15    ; FILE*
  call fputc
…

fclose to properly close the file:

…
mov rdi, r15    ; FILE*
call fclose
…

To use these functions, it was important to comply to ABI calling conventions, so I passed arguments in the appropriate registers (rdi, rsi, rdx, etc.) and preserved callee-saved registers (rbx, r12-15) across function calls.

…
    ; preserving callee-saved registers
    ; will be using r12-r15 because they survive function calls
    push rbx
    push r12
    push r13
    push r14
    push r15
…
.clean:
    pop r15
    pop r14
    pop r13
    pop r12
  pop rbx

  ret

At this point, I could take command-line arguments and produce a uniform gray image to prove that:

• the file format was correct
• pixel data was writing properly
• the output image could be opened successfully

Now, I could focus on rendering logic without worrying about input parsing or file output issues.

Pixel Traversal and Gradient Testing

My next goal was to verify that my nested row/column loops were functioning correctly before introducing floating-point math or Mandelbrot logic. I wanted to ensure that:

• each pixel is visited exactly once
• row and column indexing is correct
• the image is written in the correct orientation

Rather than writing a constant value (128) for every pixel, I computed a grayscale value based on the current row and column, then wrote the result to the image.

…
; color = (row + col) % 256
mov eax, ebx        ; eax = row
add eax, r13d       ; eax = row + col
and eax, 255        ; keep only 0-255

; fputc(color, fp)
mov edi, eax
mov rsi, r15
call fputc
…

row + col produces an increasing value across the image, and and eax, 255 keeps the value within grayscale bounds. This should result in a diagonal gradient across the image.

Floating-Point Coordinate Mapping

My next goal was to convert each pixel’s (row, col) position into a corresponding point in the complex plane because the Mandelbrot set operates on complex numbers, not pixel indices.

Map each pixel to a complex number: c = cr + ci*i

• cr (real component) is derived from the column index
• ci (imaginary component) is derived from the row index

To achieve this, I mapped integer pixel coordinates into double-precision floating-point values.

For the horizontal axis (real component): cr = real_min + col * (real_max - real_min) / (size -1)

• left edge is -1.0
• right edge is 1.0
• values in between are smoothly interpolated

To support floating-point calculations, I added constants:

• real_min: dq -1.0
• real_max: dq 1.0
• gray_scale: dq 127.5

Then, I computed the mapped value:

…
; xmm0 = real_max - real_min
movsd xmm0, [real_max]
subsd xmm0, [real_min]

; xmm1 = col as double
cvtsi2sd xmm1, r13d

; xmm0 = col * (real_max - real_min)
mulsd xmm0, xmm1

; xmm2 = size - 1 as double
mov eax, r12d
dec eax
cvtsi2sd xmm2, eax

; xmm0 = col * range / (size - 1)
divsd xmm0, xmm2

; xmm0 = real_min + ...
addsd xmm0, [real_min]

At this point, xmm0 contains the real coordinate cr in the range [-1, 1].

Before implementing the Mandelbrot algorithm, I used this mapped value to generate a horizontal gradient to verify that the mapping was correct.

To convert from the range [-1, 1] to [0, 255], I applied: color = (cr + 1.0) * 127.5

; shift from [-1, 1] to [0, 2]
addsd xmm0, [real_max]

; scale to [0, 255]
mulsd xmm0, [gray_scale]

; convert double to integer
cvttsd2si eax, xmm0

Writing the pixel:

; fputc(color, fp)
mov edi, eax
mov rsi, r15
call fputc
…

Now, I was able to confirm that integer pixel coordinates were correctly mapped into double-precision floating-point values and that floating-point instructions (cvtsi2sd, mulsd, divsd, addsd) were working correctly, which is important because the Mandelbrot algorithm will depend entirely on accurate mapping from pixels to complex numbers.

Mandelbrot Iteration

My final goal was to implement the Mandelbrot iteration itself and determine how each pixel should be colored.

For each pixel, I now have a corresponding complex number: c = cr + ci*i

The Mandelbrot set is defined by repeatedly applying z = z^2 + c starting from z = 0.

Now, I need to be able to determine if each pixel remains bounded, or if it diverges to infinity. To implement this, I represented the complex number z using two double-precision values:

• zr = real component (stored in xmm2)
• zi = imaginary component (stored in xmm3)

If |z| > 2, the point escaped, and will be colored white. If it never escaped after max iterations, it will be colored black.

• zr_next = zr*zr - zi*zi + cr
• zi_next = 2*zr*zi + ci

Inside the loop:

…
; zr_next = zr*zr - zi*zi + cr
movsd xmm4, xmm2    ; xmm4 = zr
mulsd xmm4, xmm2    ; xmm4 = zr*zr

movsd xmm5, xmm3
mulsd xmm5, xmm3    ; xmm5 = zi*zi

subsd xmm4, xmm5    ; xmm4 = zr*zr - zi*zi
addsd xmm4, xmm6    ; xmm4 = zr_next = + cr

; zi_next = 2*zr*zi + ci
movsd xmm5, xmm2
mulsd xmm5, xmm3    ; xmm5 = zr*zi
mulsd xmm5, [two]    ; *2
addsd xmm5, xmm7    ; xmm5 = zi_next = +ci

The updated values are stored back into zr and zi for the next iteration.

; updating with new values
movsd xmm2, xmm4    ; xmm2 = new zr
movsd xmm3, xmm5    ; xmm3 = new zi
…

To determine whether a point diverges, I used the Mandelbrot escape condition: |z| > 2. Instead of computing a square root (which is expensive), I used the equivalent comparison: zr*zr + zi*zi > 4

…
    ; escape check
    ; if zr*zr + zi*zi > 4.0, escaped
    movsd xmm0, xmm2    ; xmm0 = zr
    mulsd xmm0, xmm2    ; zr*zr

    movsd xmm1, xmm3
    mulsd xmm1, xmm3    ; zi*zi

    addsd xmm0, xmm1    ; xmm0 = zr*zr + zi*zi

    comisd xmm0, [escape4]    ; compare mag squared to 4.0
    ja .escaped    ; if above, it escaped
…

Coloring the pixels:

…
.not_escaped:
    ; if this label is reached, point never escaped during iterations
    mov eax, 0    ; black
    jmp .write_pixel

.escaped:
    mov eax, 255    ; white
…

So, for each pixel:

• initialize z = 0
• iterate up to N times
• stop early if escape condition is met
• assign a color based on the result

nasm -felf64 mandelbrot.asm -o mandelbrot.o
gcc -no-pie -znoexecstack mandelbrot.o -o mandelbrot
./mandelbrot 512 100 mandel.pgm