Private vs Protected

Pretty much every object oriented programming language let's you set the visibility of object members. Usually it defaults to public. That means everybody can access that member. Ofcourse often you don't want to other objects to access certain parts of your object. This is where private or protected come in to play.

Limiting members access

An object should only expose members that really need to be accessed from the outside. All other members should be "hidden". It's not just a security issue but also creates nicer foot prints to work with the code. For example IDEs only have to list the public accessable functions in their "code insight" (or whatever you want to call it) listing. The public members are the members to work with, it also makes it easier for you to find what entry points you need to use on an object (in case it's not your code).

Private

The actual outcome of this depends on the language. But usually it comes down that only the objects within the same scope can access this member, subclasses do no have access to this member. What the scope is depends on the language used, often the scope is just the object itself. In ObjectPascal however the scope is the unit file (the source code).

One of the basics of OOP is that you can extend an object and add additional functionality. And usually you can only add additional functionlity outside of the "scope" because you can either not touch the original source or you simply shouldn't.
Using private to protect members of an object will prevent an subclass to work with these members. This is often needed.

Protected

Protected is similar to private except that subclasses can always access these members no matter if they are in the same scope.

Making members protected opens a security issue for those members. Since a subclass has access to these members it's possible to work around security measures.

How to choose

I'm sure somebody already though about that, however I am not aware of a document that introduces a formal method for member accessability. If such document would exist it would be very technical for most programmers to understand, or at least it would way to time consuming. That doesn't mean it's not wise to read it and use it's methods.

Here's a simple classification scheme I think should work well enough for most people.

1. Is the member valuable to access from other classes? ⁽¹⁾
   Yes -> go to 2; No -> go to 3.
2. Is it a variable?
   Yes -> go to 2.1; No (it's a function) -> public
   1. Should this variable be read-only? ⁽²⁾
      Yes -> create a public accessor function and go to 3; No -> public
3. Does the member contain\give access to security sensitive information? ⁽³⁾
   Yes -> go to 3.1; No -> protected
   1. Is it a variable?
      Yes -> go to 3.1.1; No -> go to 3.2
      1. Should a subclass be able to write this member?
         Yes -> go to 3.3; No -> go to 3.1.2
         1. Should a subclass be able to read this member?
            Yes -> private and create a protected accessor method; No -> private
      2. Does this member give readonly access?
         Yes -> go to 3.2.1; No -> go to 3.3
         1. May a subclass read this data? ⁽⁴⁾
            Yes -> protected; No -> private
      3. Does the new value need to be validated? ⁽⁵⁾
         Yes -> private and create a protected method that will validate and set the private method; No -> protected

Only in a few cases the member will become and isolated private member (e.g. unable to read\write to the member).

1.

This is quite a difficult question, is the member for internal use only? When is something for internal use only? Things like buffers and pointers are usually for internal use only. Maybe somebody else can shed some light on this question.

2.

Very often you want to be able to give other classes access to certain variables, like the current position of a parser, but you don't want them to be able to modify the value. For this you should create an accessor function getMyVariable() that will simply return the current value of the variable.

3.

When is something security sensitive? Things like keys and passwords and sensitive. But it's not limited to authentication related information, sometimes things like an URI could be sensitive. Things that might need to be validated before they are used are security sensitive. For example in games you might want to give a user the freedom so configure some client side things. However, these things could be abused for cheating. So the settings need to be validated before they are set.

4.

In only a very few cases you want to prevent a subclass from reading certain variables. It's pretty much only the case for keys and passwords. Even though a subclass might have set the value via an other method it still doesn't mean it should be able to read a password.

5.

In some cases you want to give a subclass the right to set a new value, but you also want to validate the new value to meet certain security measures. For this you need to make sure the subclass has to use your method to set the new value and not be able to override it with it's own faulty method. Most languages provide the means to do this (UnrealScript allows you to mark a function as final, thus preventing it to be overwritten by a subclass). There is a trick to do this. Simply make the member private so it can't be accessed from the subclass and create protected accessor methods (get and set). In your set method you will validate the new value and pass it to the protected member. This way subclasses will have to use the parent set method in order to write the new value.

 //pseudo code

 var private string SensitiveInfo;

 protected function bool SetSensitiveInfo(string newValue)
 {
  if (newValue doesn't meet constraints) return false;
  SensitiveInfo = newValue;
  return true;
 }

 protected function string GetSensitiveInfo()
 {
  return SensitiveInfo;
 }