O_APPEND by itself would imply O_RDONLY (because it's 0), not O_WRONLY or O_RDWR. Opening read-only for append isn't usually useful. As in C, you need to OR together multiple flags to get a file descriptor you can use to write at the end of the file, even if its length is changed by something other than than your writes.
For appending to a file use the O_APPEND = 0x400 flag as int flags parameter in rsi. So you were close: it's a value of hexadecimal 400 and not decimal 400 as you attempted.
Here is a list of the raw int flags values in hexadecimal:
O_ACCMODE = 0x3
O_APPEND = 0x400
O_ASYNC = 0x2000
O_CLOEXEC = 0x80000
O_CREAT = 0x40
O_DIRECT = 0x4000
O_DIRECTORY = 0x10000
O_DSYNC = 0x1000
O_EXCL = 0x80
O_FSYNC = 0x101000
O_LARGEFILE = 0x0
O_NDELAY = 0x800
O_NOATIME = 0x40000
O_NOCTTY = 0x100
O_NOFOLLOW = 0x20000
O_NONBLOCK = 0x800
O_RDONLY = 0x0
O_RDWR = 0x2
O_RSYNC = 0x101000
O_SYNC = 0x101000
O_TRUNC = 0x200
O_WRONLY = 0x1
So this should work:
mov rax, 2
lea rdi, [rel path]
mov rsi, 0x441 ; O_CREAT| O_WRONLY | O_APPEND
mov edx, 0q666 ; octal permissions in case O_CREAT has to create it
syscall
mov r8, rax ; save the file descriptor
I added the three values to create the file if it doesn't exist or open it in write_only mode if it does exist. You could define them as assemble-time equ constants so you could actually write mov esi, O_CREAT| O_WRONLY | O_APPEND in your source.
Also, make sure to supply a value for the 3rd arg (permissions) any time you include O_CREAT. Otherwise the file could get created with random garbage for its permissions, including possibly setuid + exec. Normally you pass octal 666 (NASM 0q666) and let the user's umask knock off write permission for other and optionally group.
If you don't actually want open to create it if it doesn't already exist, omit O_CREAT. Then it will return -ENOENT.
Also is there a way to not specify the length of the string in rdx, but instead have the string be terminated by a /0?
In short: No.
Create a strlen function and pass the result in rdx. Kernel system calls take data to be read/written on file descriptors as buffer + length, only pathnames as C implicit-length strings.