I'm writing some procedures in x86 assembler that modify ZF as a means to return boolean values, so I can do something like this:
call is_value_correct
jz not_correct
I'm wondering if this is considered bad practice since some coding standards say that simple values should be returned in the AX register.
Do it if it makes your code run faster and/or be smaller overall. It's not bad practice.
One of the advantages of writing in asm by hand is being able to use custom calling conventions functions, even when they're not private helper functions or macros. This includes being able to "return" multiple values in different registers, and basically do whatever you want.
As with any custom calling convention, all you need to do is document it with comments. Here's an example of how you might write such comments, with a specific and intentionally non-standard set of things.
# inputs: foo in EAX (zero-extended into RAX), bar in EDI
# pre-requisites: DF=0
# clobbers: RCX, RDX, AL (but not the rest of EAX)
# returns: AL = something, ZF = something else
my_func:
...
setc al
...
something that sets ZF
ret
If you're willing to sacrifice efficiency for style or readability, you probably shouldn't be writing in asm in the first place in 2018 when compilers are capable of generating good asm most of the time, and you rarely need to write your own boot sector or whatever. (i.e. performance is the main use-case left for hand-written asm, and it's only appropriate if you're going all out for performance.)
Yes it's possible for hand-written asm to become an unreadable / unmaintainable mess, but if done carefully when it has a reasonable semantic meaning, this optimization won't make your code a mess.
There is even precedent for doing this: x86-64 OS X system calls use CF as the error/no-error status, separate from the rax return value. Unlike Linux where errors are indicated by RAX return values from -4095 to -1, although otherwise they use the same x86-64 System V ABI / calling convention.
Some DOS int 0x21 system calls and PC BIOS int 0x10 functions have a similar flag-return system. It lets the caller branch on error, so it saves code-size (a test or cmp) and avoids needing in-band signaling of errors.
BTW, in inline assembly, not writing a whole function, you can and should output something in FLAGS instead of wasting instructions materializing a boolean in a register, especially if the compiler is just going to test it again. Condition-code output constraints are supported since GCC6, see this answer for an example.
But it's not easy to safely call a function from inside an asm statement if you want to use C to call a function with a custom calling convention. You need to tell the compiler that every call-clobbered register is clobbered. Including st0..7, mm0..7, x/y/zmm0..31, and k0..7 (if AVX-512 is supported, otherwise the compiler won't even know those reg names). Or only the ones your hand-written asm function actually clobbers, if you want to commit to maintaining the constraints in sync with the function's actual implementation.
And call itself does a push which steps on the red zone. See Calling printf in extended inline ASM for an example that should be safe.
