The calling FAX machine starts sending a (broken) tone as soon as it finishes dialing. The switch answers the line and listens for this tone. If the tone is there, it routes the call to the FAX. If it isn't it rings the voice instrument.
Of course by this time the caller is paying for the call so it's good manners to have an answering machine on the line so that they won't be left paying to listen to a ringing signal.