It uses regular expressions. The caret symbol means the condition must match the beginning of the string in your ^YandexBot example. So that means it should have matched ^YandexBot with YandexBot/3.0 unless the user agent started with something else. Yes, the asterisk works like you're thinking, but it should have matched the way it is currently. Something certainly is not right.
If you've changed the code please post the new .htaccess code you're using. Also, to test have you tried placing that .htaccess file in the account's public_html directory rather than in the home directory? Maybe it's not working for some reason where it's at currently, file permissions or changes to apache's config.