[HighLow21]

Hi everyone,

A post by wyman about the possibility of changing the skill-level breakdowns in the forums (currently "B/I" and "A/E") has prompted me to start thinking about the self-rating system here on BBO. I think many people will agree that it's better than nothing but, as it currently stands, it's often insufficient as implemented and actually used.

I'd like to begin a discussion about ways to improve it --> not that such a discussion will necessarily lead to actual changes, but more to flesh out some possible ideas for changes and get a sense of what others think of the system and how it could be improved. If the discussion is thoughtful and productive, perhaps it will give the BBO management some incentive to consider implementing some of the recommended changes.

The Problem.
To me there are 3 significant problems with the system that could potentially be fixed (I encourage people to include their own perceived problems in this thread):
(A) The explanation for self-assigning the skill rating is far too simple and compares things that are not really comparable. (I will attach the explanation in a follow-up post to this thread.)
(B) The explanation, as written, leads to far too large a percentage of the BBO population being lumped together in "Intermediate" and therefore is less helpful than it could be.
(C ) Because the system is self-assigned and is not corroborated by any metrics of actual experience or ability, there are many people who self-rate very inaccurately.

I think (A) and (B) are actually very easy to fix or to improve upon. I think (C ) opens a massive can of worms, but still can be improved upon. It just would require a lot more work, in my estimation.

Problem (A): Explanation is too simple.
I've attached the explanation for choosing a rating in a follow-up post to this thread. It only takes into account one dimension of a player's skill and experience levels, and across different ability levels, the dimensions taken into account are not comparable. For example:
- the definition of a "Beginner" is given in terms of calendar length of experience;
- the definition of "Intermediate" is based on a broad definition of skill RELATIVE to other players;
- the definition of "Advanced" is based upon performance in two specific environments (clubs and minor tournaments).
This only takes into account one aspect of a player's overall skill and experience, and across different categories, the aspects are not comparable.

Here is one suggestion I propose to improve this:
(A1) Develop a self-assessment "matrix". The column headers of the matrix represent different dimensions of a player's game. The row headers of the matrix correspond to different skill designations. Explain that most players will show signs of different ability levels for different dimensions of the game. Ask the player to self-rate with the skill designation that seems to be the closest overall match.

For the row headers I propose the following:
- Bidding system comprehension
- Bidding skill
- Declarer play skill
- Defensive play skill
- Length of experience
- Lifetime # hands played
- Typical performance results
- Tournament participation

For the Intermediate row, as an example, the descriptions I would include in the columns would be something like this:
- Bidding system comprehension: "Strong or complete understanding of one system (e.g., SAYC, 2/1, Precision) and some knowledge of other systems. Have played variations of some conventions."
- Bidding skill: "Ability to re-evaluate hands under some conditions. Capable of making some good judgment calls not dictated by system."
- Declarer play skill: "Ability to count winners and losers. Ability to assess different lines of play and compare some of them. Some experience with advanced play concepts, such as endplays and squeezes."
- Defensive play skill: "Understands basic signalling. Has a decent sense of when to switch suits and which suits to switch to. Understands the concept of passive vs. active defense. Can assess some of the pros and cons of different opening leads depending upon the auction."
- Length of experience: "Typically 1-10 years of experience."
- Lifetime # hands played: "Typically 10,000-100,000 lifetime hands."
- Typical performance results: "Above average in the MBC; average to above average in minor tournaments; interest in developing as a player and proceeding to more challenging levels of play."
- Tournament participation: "Depends, but typically active in minor tournaments with the possible occasional entrance in higher level tournaments."

Problem (B): Too Many Intermediates.
As we discussed, the current system is set up such that probably 90% of BBO players fit into this category. Two possible solution ideas:
(B1) Redefine Intermediate bracket endpoints within the current context. For example, make the bottom level something like "2+ years of experience" and make the top level something like "some success in MBC, clubs, or minor tournaments."
(B2) Proceed as in (A1), above. Further flesh out ranges of ability across all dimensions of play at all levels. Potentially tighten the definition of Intermediates, allowing some players to flow back into Beginner and others to flow into Advanced.
(B3) Add a new class between Beginner and Intermediate. Call this category "Improving" or "Solid" or "Average" or some such thing. Broadly, have this category take on the characteristics of what would probably constitute, roughly, the bottom half of current Intermediates.

Other ideas appreciated.

Problem (C ): Self-Rating is unsubstantiated and often inaccurate.
I think this problem speaks for itself, but I foresee a broad spectrum of ways to improve this.

Here are some ideas for "quick" fixes:
(C1) Give an indication of # of lifetime BBO hands played in the player profile. (The best proxy we have currently is # of logins and signup date.)
(C2) Give an indication of scoring averages attained by the player, either lifetime or over the past month or so.
(C3) Give an indication of the player's level of participation in tournaments. (# of tournaments played, recent results, etc.)

Here are some ideas for more difficult fixes:
(C4) Have the player rating tied directly to the aforementioned metrics. (This leads to a whole host of potential problems as discussed in a separate thread, where some of the behaviors of players on a different site, which does this, are designed explicitly to manipulate or protect their rating. This can be a huge problem.)
(C5) Have a battery of tests available for players to take to "qualify" for various levels of ability. Make it so that self-assigned ratings of some levels are unavailable to people who have not passed these qualifying tests.
(C6) Develop a "wizard" that a player can use to review past decisions (bidding and play) and assess the player's skill. (This is a huge undertaking but worth its weight in gold and highly marketable if done well.)

Any thoughts appreciated.
-Tate

[hrothgar]

Do we need a FAQ?

[diana_eva]

Did you check the pinned thread about ratings on BBO, Tate?

[phil_20686]

I can think of a few obvious problems:

(1) A large fraction of people are incapable of rating themselves. By this I mean, that if you have never played against genuine experts, you will not realise quite how deep bridge is, and how much better it is possible to be. I would guess that this includes a huge fraction of those players who win regularly at their local club but do not play much tournament bridge.

(2) A large fraction of people do not wish to rate themselves accurately. Either they want an ego boost, or they want to play against good players. An over accurate ratings system leads to stratified games, which is not in everyone's interest. Inevitably people will game the system to reach the outcome they want, rather than the outcome that was designed.

(3) Generally good players on BBO know other goodplayers who play on BBO so they only play with friends, so self rating is not much of a problem.

[HighLow21]

diana_eva, on 2012-March-13, 12:16, said:

Did you check the pinned thread about ratings on BBO, Tate?

How about this: I am going to do so now, and afterwards will edit my OP to reflect anything relevant. Sound good?

[CSGibson]

This has been talked about ad nauseum in the forums every few months. Who cares what others rate themselves? Really?

Last week I had someone I was playing with for the first time at a club tell me that they are sometimes good and sometimes bad. I then played with them and discovered that, in all likelihood, they were always bad, and weren't good enough to know it. Did I tell them that? No. I just made a mental note and moved on. I do that on BBO, too, except they are kind enough to allow me to make actual physical notes for players who have, in my opinion, inaccurate ratings.

If you want a rating system with some fundamentals behind it, pay to play on OK bridge. Otherwise, play free bridge at BBO and deal with it.

[HighLow21]

phil_20686, on 2012-March-13, 12:28, said:

(1) A large fraction of people are incapable of rating themselves. By this I mean, that if you have never played against genuine experts, you will not realise quite how deep bridge is, and how much better it is possible to be. I would guess that this includes a huge fraction of those players who win regularly at their local club but do not play much tournament bridge.

Agreed with this, but if the Matrix idea (A1, above) is implemented thoughtfully, and people actually use it as honestly as they are capable of using it, it could help and couldn't hurt in my opinion. Plus, I see tremendous value from trying to separate the 90% or so who would fall into 'Intermediate' under the current regime.

phil_20686, on 2012-March-13, 12:28, said:

(2) A large fraction of people do not wish to rate themselves accurately. Either they want an ego boost, or they want to play against good players. An over accurate ratings system leads to stratified games, which is not in everyone's interest. Inevitably people will game the system to reach the outcome they want, rather than the outcome that was designed.

Of course not. In any self-rating system, intentional misuse cannot be avoided. My thesis is simply that the current system and explanation of it aren't particularly good, and that it's not as helpful as it could be to people who WANT to self-assess accurately.

phil_20686, on 2012-March-13, 12:28, said:

(3) Generally good players on BBO know other goodplayers who play on BBO so they only play with friends, so self rating is not much of a problem.

I agree that there are ways around the problems inherent in the current system. That's not, in my opinion, a valid reason for failing to entertain ideas for improving the current system.

[hrothgar]

HighLow21, on 2012-March-13, 12:55, said:

Agreed with this, but if the Matrix idea (A1, above) is implemented thoughtfully, and people actually use it as honestly as they are capable of using it, it could help and couldn't hurt in my opinion. Plus, I see tremendous value from trying to separate the 90% or so who would fall into 'Intermediate' under the current regime.

If wishes were horses we'd all be riding...

[HighLow21]

CSGibson, on 2012-March-13, 12:49, said:

This has been talked about ad nauseum in the forums every few months. Who cares what others rate themselves? Really?

(1) I do, and a lot of people I know do. In fact, I would posit that almost everybody who thinks about it cares about it, but many who would care MORE about it have dismissed it entirely because it's so inaccurate. That's precisely why they don't care about it. That, anyway, is my opinion.
(2) Really.
(3) I haven't seen the other posts and I am sure this has come up many times. I'm not interested in re-inventing the wheel here, but I had some ideas and thought "It's better to put them out there and see whether they gain any traction that to sit and grumble about the problems in a system that I find inherently fixable."
(4) The idea came up specifically today in a discussion about the possible splitting of some skill level-based forums here (specifically, the B/I and A/E split, and whether it'd be better to do something else, such as B, I, and A/E). My thought is: if we're discussing this split, why not discuss improving the self-rating system as well?

CSGibson, on 2012-March-13, 12:49, said:

If you want a rating system with some fundamentals behind it, pay to play on OK bridge. Otherwise, play free bridge at BBO and deal with it.

You are entitled to your opinion Chris, but in all due respect (and I do mean this, I'm not being sarcastic): so am I.

[HighLow21]

Here is the description available for self-rating according to BBO. This is available from Help | The Rules of this Site in the web-based client:

Quote

There is an area in your user profile where you can enter information about your level of play. It is not important how well you think you play since most bridge players are unable to assess their own skills with any accuracy. What matters is your level of experience and the degree of success that you have had in live competitive clubs and tournaments. Here are some examples of terms that you can use and how they should be interpreted:

Novice - Someone who recently learned to play bridge
Beginner - Someone who has played bridge for less than one year
Intermediate - Someone who is comparable in skill to most other members of BBO
Advanced - Someone who has been consistently successful in clubs or minor tournaments
Expert - Someone who has enjoyed success in major national tournaments
World Class - Someone who has represented their country in World Championships

If you disagree with another member's assessment of his or her skills, it is completely inappropriate to mention this to them (via chat for example). All members should strive to give an honest approximation of their skill level. This is very much in the best interest of everyone who plays on BBO.

If you would like to join a table to which players of a certain level have been invited, you should follow the above guidelines in order to judge whether or not you would be welcome.

[HighLow21]

diana_eva, on 2012-March-13, 12:16, said:

Did you check the pinned thread about ratings on BBO, Tate?

Here is the thread Diana is referring to:

Rating Players: Basic Theory

It is in the general BBO Discussion forum, not this one. It is an excellent read from what I've looked at so far, and if I see any reason to update my original post after reading it all, I will definitely do so.

[CSGibson]

Every "solution" you proposed is fraught with problems which create a worse environment than is currently there.

Everyone has told you this. You haven't absorbed it. Please, repeat it to yourself 1000 times to get it stuck in your head.


IMP AVERAGES ARE MEANINGLESS

I regularly practice against top level players (who, as partners, finished 2nd in a Blue ribbon) with my regular partner. I also play against and with randoms. Imp averages treat those boards the same, when obviously they are not. If I only played against top level players, I'd expect an imp average of around 0, or slightly negative. In your proposed rating "fix" that would identify me as a neutral or bad player. That's not the case BECAUSE IMP AVERAGES ARE MEANINGLESS without context.

The rest of your solutions have similar problems. Having those ratings would mean that mentors get punished in their ratings, and that people will change how they play bridge online in order to "win" the ratings game. That, in turn, will hurt the overall BBO experience in my opinion.

It ain't broke, Tate. I understand that sometimes people get frustrated over what they perceive to be inaccurate ratings, but those people are always going to get frustrated playing bridge with strangers - their frustration is not a reflection of ratings, so much as their own personality. They have to get over their personal issues, realize its just online bridge, and move on to a different game if they aren't having fun at the table they are at.

[HighLow21]

CSGibson, on 2012-March-13, 13:14, said:

<snip>
IMP AVERAGES ARE MEANINGLESS
<snip>

Chris I recognize that a results and performance-based rating system (i.e., not one that is self-assigned) is fraught with dangers and a lot of work to implement.

That is why I clearly separated it out as Problem (3).

Problems (1) and (2) if handled right, in my opinion, could be done (a) without many new problems, (b) without very much work, and (c ) with substantial positive upside potential for the BBO community.

[wyman]

HighLow21, on 2012-March-13, 11:58, said:

A post by wyman about the possibility of changing the skill-level breakdowns in the forums (currently "B/I" and "A/E")

Just to be clear, my post, which happened to express the opinion that we could do with a "total newbie" forum, was primarily intended to encourage beginner lurkers (who might be offput by the level of advice in B/I) to go ahead and post anyway.

[HighLow21]

wyman, on 2012-March-13, 13:31, said:

Just to be clear, my post, which happened to express the opinion that we could do with a "total newbie" forum, was primarily intended to encourage beginner lurkers (who might be offput by the level of advice in B/I) to go ahead and post anyway.

OK that's a very good point. So your suggestion is to have (a) "Novice" or "Newbie" forum, (b) B/I, and (c ) A/E, correct?

Doing so might provide some additional benefits, as well, and more broadly, the idea of realigning the forums in (1) precisely your suggested way or (2) some similar derivative of that way couldlead to other benefits as well.

As a comment to your thread pointed out, self-rating comes into play here... after all, how does one know which forums to read and post in, unless one has a fairly good idea of where they stand? Hence the inspiration for me to post about my thoughts on the current rating system-->followed by some suggestions that are open to criticism, refinement, discussion, etc.

[HighLow21]

After reviewing some of the replies to this thread I want to be clear here: I am suggesting 3 separate problems, as I see them, with the current BBO rating system. They are presented in increasing order of complexity, in terms of the cost of implementing a solution to them, and the magnitude of the potential dangers of such implementations.

(1) The current system is far too simple, in terms of its explanation, for most people to use accurately. Let's improve the documentation of that system.
(2) The current system results in far too many people being most accurately assessed as "Intermediates." This limits the usefulness of the system. Let's consider revising the core system so that not as many people would most accurately fit into "Intermediate."
(3) Self-rating alone, no matter what improvements can be made upon it, is fraught with problems. Can we consider alternative options to this.

Note that (1) and (2) involve leaving the current system essentially unchanged, but improving upon it at the margins. In my view: small cost, small set of new problems, vast potential improvement. Please reread this if it isn't clear.

I never expect many possible solutions to (3) to get implemented here, but maybe there's a compromise that will make some people happy. Hey, if we're talking about a "wish list," not a "to-do list," it can't hurt to discuss it, right?

[hrothgar]

HighLow21, on 2012-March-13, 13:24, said:

Problems (1) and (2) if handled right, in my opinion, could be done (a) without many new problems, (b) without very much work, and © with substantial positive upside for the BBO community.

I think that you're wrong on all fronts...

Let's assume a perfect world in which the self rating system was accurate. I really don't see much upside to the community at large. I might be an outlier, but "skill level" really doesn't factor into my decisions about where / when I am going to play.

I play with folks much better than me...
I play with folks much worse...

And I enjoy both sets of games (even though I secretly suspect that I am occasionally invited to games for comic relief)

Over the years, I've given a fair amount of thought to this topic.

Here's how I think that things could (should ???) be addressed. (Not sure if its worth the effort though)

1. Start by creating a large corpus of bridge questions:

Some might be related to bidding
Some might be related to play

2. Get a whole bunch of people to take the quiz

Run the results through a clustering algorithm...
Identify obvious clusters in the population of bridge players

3. Use some classification algorithms to collapse the set of bridge questions down into a small number with the maximum discriminating power.

At this point in time, you can give players the option to take a short quiz

The quiz will identify that you fall into Cluster "X", that you are compatible with clusters "Y" and "Z", but you should steer very clear of clusters A, B, and F...

[In case anyone cares, this approach is essentially modeled after the methods that dating sites use for matching people. I'm just tweaking it for bridge players rather than ...]

[mgoetze]

(A) and (B) are not the problem, and BBO management is not interested in fixing ©. End of discussion.

[billw55]

Quote

Novice - Someone who recently learned to play bridge
Beginner - Someone who has played bridge for less than one year
Intermediate - Someone who is comparable in skill to most other members of BBO
Advanced - Someone who has been consistently successful in clubs or minor tournaments
Expert - Someone who has enjoyed success in major national tournaments
World Class - Someone who has represented their country in World Championships

My nitpicky gripe about this system is that, semantically, "beginner" ought to be a lower skill level than "novice".

Regarding the actual discussion: among the various solutions, I think that implementing a true rating system is most likely to be beneficial. Beneficial enough to warrant the resource investment by ownership? Probably not, but perhaps. On the other hand, greater detailing of the self-rating options would take comparitively less investment .. but also is less likely to be useful, IMO.

[Vampyr]

CSGibson, on 2012-March-13, 12:49, said:

If you want a rating system with some fundamentals behind it, pay to play on OK bridge.

Seriously, it's about $100/year. Why did you think this was a bad idea, OP?