PEM_read_RSAPublicKey() crashes in main application code but works fine in sample code. How can i fix this issue?

Question

I am writing a c/c++ code for Win32 Dynamic_link Library in visual studio 6. Actually i am using libcryptoMD.lib for implementing RSA encryption using pem file and ppk file. I'm able to do this encryption in sample code but PEM_read_RSAPublicKey() crashes in actual application code.

I have already tried setting all the Project Settings of actual application code to sample project as there could be some conflicts in dependency libries of actual application code. But to my surprise, sample project worked successfully and actual application failed. I also searched for why this function could have crashed in actual application code, but could not find any useful answer. This function needs filepointer of public key pem file and RSA pointer which are absolutely fine.

RSA * create_RSA(RSA * keypair, int pem_type, char *file_name) {

    RSA   *rsa = NULL;
    FILE  *fp  = NULL;

    if(pem_type == PUBLIC_KEY_PEM) 
    {
        fp = fopen(file_name, "rb");

        PEM_read_RSAPublicKey(fp, &rsa, NULL, NULL);

        fclose(fp);

    }
    else if(pem_type == PRIVATE_KEY_PEM) 
    {
        fp = fopen(file_name, "rb");
        PEM_read_RSAPrivateKey(fp, &rsa, NULL, NULL);
        fclose(fp);
    }

    return rsa;
}

PEM_read_RSAPublicKey(fp, &rsa, NULL, NULL); crashes showing

OPENSSL_Uplink(6F8D1880,08): no OPENSSL_Applink in actual application code.

You never bother to check if your `fopen()` calls actually work. — Andrew Henle, Oct 02 '19 at 14:38

score 0 · Answer 1 · answered Oct 02 '19 at 16:18

It's not clear if you're trying to use two levels of DLL here -- i.e. do you have an EXE that calls a DLL that contains your code that in turn calls the OpenSSL DLL(s)? If so that doesn't work; the kludge OpenSSL uses for 'uplink' in order to do file I/O can only handle an EXE that (directly) calls the OpenSSL DLL(s). Your alternatives are: statically link OpenSSL (into your DLL); or use the BIO interface instead so that file I/O is handled (entirely) at the OpenSSL level, or do the I/O (here input) yourself and use a memory BIO for OpenSSL; see my answers to this similar Q.

If you are calling from the EXE to OpenSSL DLL(s), it should work if you compile and link applink.c into your EXE as documented (tersely) on the man page, or you can choose the alternatives above if you prefer.

In addition: as Andrew commented, you should be checking for errors on the fopen or other I/O, and also on the PEM_read routines -- they can fail. Also, PEM files are text and don't need the b flag, even on Windows. And using the lower-level RSAPublicKey format, instead of the [RSA_]PUBKEY format, is very unusual; are you sure that's what you want?"

Thanks dave_thompson_085. Linking openssl dll directly to main exe and adding PEM_read_RSAPublicKey() in main exe helped me. — user12154135, Oct 25 '19 at 10:32

PEM_read_RSAPublicKey() crashes in main application code but works fine in sample code. How can i fix this issue?

1 Answers1