but I don't know how to convert the set of hex digits the user inputs into memory into the format required for the FAC
I have a strong feeling that this is not about hex, but binary integers, 16 bit or more.
I know about FRMNUM ($DD67) and PRNTFAC ($ED2E), but I can't find the subroutine that Applesoft uses to take a decimal number and convert it to the six-bit exponent-mantissa format.
You're already on the right track. For example GIVAYF ($E2F2 *1) converts a 16 bit integer held in A/Y (High/Low) into a floating point number in FAC.
; Converting a 16 bit value to FAC ; IN: - ; OUT: $1234 as FP value in FAC LDA #$12 ; 16 Bit $1234 in A/Y LDY #$34 JSR GIVAYF ; Convert it to FAC ... Of course this is less handy if your numbers are not straight integers, but some format of your own - which brings us to what might bet the primary issue:
Where is your data is originated?
Is it from some internal storage, or user input. If it's from user input why not skipping the task of reading a number, turning it into some two part integer and then convert it to FP and use the Applesoft input routines directly?
Which brings us back to the already mentioned FRMNUM, which takes a numeric input from where TXTPTR ($B7) points to and evaluates it into a value in FAC. Of course FRMNUM is a rather complex beast, doing all sort of things, including expression evaluation (*2) not just for numbers but strings as well.
A more down to the minimum task version would be FIN ($EC4A), which reads a ASCII string and turns it into FAC. FIN expects again TXTPTR to point to the string to be converted, but as well the first character already read into A. Numbers read are in the format of [sign]integer[.integer[E[sign]integer]]. Anything not fitting that format will terminate, so best use a colon or zero byte. It returns with the number in FAC, or an error. Calling it could look like this:
; Converting a numeric string to FAC ; IN: String in MYBUF ; OUT: FP value in FAC LDA #<MYBUF ; Lets point to the number we've inputted STA TXTPTR LDA #>MYBUF STA TXTPTR+1 JSR CHARGOT ; Get the first byte into A JSR FIN ; Convert it to FAC ... MYBUF DC '123.456E-7',':'. On a side note, it would be great if you could not only comment your source (*3), but as well add it not as picture but text. This does not only increase readability, but helps others a lot to reply with code sniplets you might love to get.
*1 - I always loved that address, it's o easy to remember: Enteger to Float too (yeah, I know, silly, can't help)
*2 - Sadly not as convenient as some other BASICs VAL()functions, as it ecpects the operators to be tokenized. Which of course could be done, essentially turning you'r calculator into a single line program, but that's a different story.
*3 - I would consider some header describing input/output/task a bare minimum to each and every function. Keep in mind, you want to read your code in 10 years from now as easy as it is today - without a need to dissassemble your own thoughts.