I’m afraid the blog homepage CANNOT be set hidden from the public. When you set it under Settings >> Reading >> either as a Static Post Page or Homepage, it will be made visible to public.
Though, you should be able to protect posts and categories using OptimizeMember so that when someone clicks the ‘Read More’ link for a post they will then be asked to login first before gaining full access to a certian protected post or be redirected to your membership options page.
To protect posts/categories using OptimizeMember, please go to OptimizeMember panel >> Restriction Options >> Post Level Access Restrictions OR Category Level Access Restrictions. You should be able to read detailed information on the said feature on each panels.