I created an unmanaged C++ DLL that I'd like to provide to a 3rd-party. The DLL will be distributed publically but I don't want anyone else to be able to call methods in the DLL. What's a good way to restrict access?
FWIW - the DLL will be used on both Mac and Windows.
Ultimately, you can't stop somebody who's determined, but you can do a few things to make life difficult for unauthorized users.
The obvious would be to encrypt (at least most of) the code in the DLL. Have a single entry point that the user has to call to get (for example) a table of pointers to the real functions. When the entry point is called, a key has to be supplied and that is used to decrypt the rest of the content, and then the pointers are returned. If the user supplies the wrong key, they get pointers, but the data won't be decrypted correctly, so when they try to call it, it doesn't work.
That's not foolproof by any means. Just for an obvious example, if somebody runs the authorized program under a debugger they can see what key is passed, and simply re-use it. If you want to make life a bit trickier for an attacker, you can do something like having the DLL do a checksum on the executable, and use that as the key (and the user will have to embed some data to get the checksum to come out right). That can still be figured out, but it will at least take some work.
Edit: The exact encryption doesn't matter a lot. I'd probably use AES, simply because it's easy to find an implementation and it runs pretty fast, so there's not much reason to use something else.
As far as how the calling code would work, it would be something like this:
struct functions { rettype1 (*func1)(args1); rettype2 (*func2)(args2); rettype3 (*func3)(args3); // ... }; void entry_point(key_type key) { functions f; decrypt(data_block); f.func1 = data_block.offset1; f.func2 = data_block.offset2; // ... return f; } In the calling code you'd have initialization something like:
functions f = entry_point(myKey); and then calling a function would be something like:
f.whatever(arg1, arg2, arg3); In essence, you're re-creating a small piece of object oriented programming, and having your entry point return an "object" of sorts, with the DLL's real functions as "member functions" of that "object".
As far as getting the checksum goes, I haven't thought through it in a lot of detail. Basically just just want the address where the executable is stored. You can walk through modules with VirtualQuery, or you can use GetModuleHandle(NULL).
key parameter. An attacker merely needs to set a breakpoint on entry_point and look at what was passed as the key parameter. They can then copy that key to their progarm.Anyone with access to the dll will be able to inspect it, load it, and get pointers to functions in the dll. I'm not sure you strictly mean dll though as you say:
FWIW - the DLL will be used on both Mac and Windows.
Dlls aren't a Mac thing--they are a Windows specific implementation of a runtime loaded/linked library. So you're real question, I'm assuming, must simply be about limiting access to a library for only certain 3rd parties. To achieve your goal, some other options may be more appropriate:
Instead of a dll, compile everything into a single monolothic exe. Run that exe as a service or a second process on the box. Have the 3rd party connect to it through sockets/named pipes/whatever other forms of interprocess communication. Authenticate the users of the service so that only the 3rd party software with the right credentials can access the service. Everyone else gets denied access.
Create an account for the 3rd party application. The 3rd party application can only run when that account is logged in. Place the dll (or dylib in the case of Mac) in a folder only accessible to that account. Noone else has access to that location, so no other applications can run it barring those run as administrator.
