c++ inline asm get OF register in flags register

Question

I want use OF register to check for overflow.
I use this code :

auto GetFlags() {
    unsigned long long flags;
    int num = INT_MAX - INT_MIN;    // Test
    asm (
            "pushfq\n\t"
            "popq %0\n\t"
            : "=r" (flags)
            );
    return (flags >> 11) & 1;
}

The variable num is used to make OF register 1. But use many times cout << GetFlags() << endl; only have 0.
I want to know the reason for the error, whether I need to write the subtraction into asm ？ or the location of the obtained of register is wrong ？

translater：gcc 12.2.1
OS: Linux Arch 6.2.12(X64)
IDE: clion 2023.1.2(debug_x64)

Thanks.

Why does the title say C++ and the tag say C? Which one of those two different languages are you using? — Eric Postpischil, May 01 '23 at 18:42
In any case, I would not expect testing the overflow flag this way to be useful. There is no assurance the compiler will put the given assembly code in the program immediately after the arithmetic instruction that does a calculation for C or C++ code. And, with constants, as shown in this example, the calculation will likely be done at compile time, so there will not be any arithmetic instruction for it in the final program. — Eric Postpischil, May 01 '23 at 18:44
What are you actually trying to do? There are other questions on [detecting overflow](https://stackoverflow.com/questions/3944505/detecting-signed-overflow-in-c-c). If that is not what you want to do in this case, explain what you want to do and how it is different from detecting integer overflow as in the other question. — Eric Postpischil, May 01 '23 at 19:34
@EricPostpischil Thank you for your reminder, this is my negligence, the label has been modified, the reason is: after editing the question, the label "C++" automatically appeared, I thought this label had been selected by default, but when I want to issue a question , prompted me to need a label, I chose "C", at this time I think my label is "C++" and "C", not just "C" — Ne C, May 02 '23 at 09:35
You are correct, for the calculation of the constant, the result will be given at compile time, which I overlooked. @EricPostpischil — Ne C, May 02 '23 at 09:41
I'm trying the "isLessOrEqual" function of data_lab in csapp, I implemented it using code like this: int num = x + (~y+1) + (~1+1); int flag = num & (1 << 31); return !!flag; This code passes btest, but it has a problem that it doesn't return correct result on overflow, so I want to use (OF)^(!!flag) ，It's not clear to me how to add code formatting to the reply, so sorry for the formatting errors. @EricPostpischil — Ne C, May 02 '23 at 09:48

score 1 · Accepted Answer · answered May 01 '23 at 18:48

Some issues ...

int num = INT_MAX - INT_MIN; is calculated at compile time. So, it generates a mov instruction [which does not set any flags] instead of sub.
Under optimization, the num code is elided because the compiler thinks the variable is unused.

Here is the corrected code. It is annotated with the bugs and fixes:

#include <climits>
#include <iostream>

// NOTE/FIX: we must force GetFlags to do a real sub instruction
// even though these values can be seen, the compiler can't convert them to
// constants
#if 1
int imax = INT_MAX;
int imin = INT_MIN;
#endif

unsigned long long
GetFlags(void)
{
    unsigned long long flags;

// NOTE/BUG: this evaluates to a compile time constant of -1
#if 0
    int num = INT_MAX - INT_MIN;        // Test
#else
    int num = imax - imin;      // Test
#endif

  asm(
    "\tpushfq\n"
    "\tpopq %0\n"
    : "=r" (flags)
// NOTE/FIX: we must tell compiler _not_ to elide num calculation above with -O2
    : "r" (num)
    );

    return (flags >> 11) & 1;
}

int
main(void)
{

    std::cout << GetFlags() << "\n";

    return 0;
}

In the code above, I've used cpp conditionals to denote old vs. new code:

#if 0
// old code
#else
// new code
#endif

#if 1
// new code
#endif

Note: this can be cleaned up by running the file through unifdef -k

c++ inline asm get OF register in flags register

1 Answers1