Cautious Optimist - The Confidence Game: Projecting Performance (Part 1)

 

The organizing framework

In the two previous posts, I've argued for the following claims that together constitute the framework I bring to my analyses of baseball organizations and to the Mets in particular: 

 (1) A successful organization must manage risks rationally. Doing so requires (a) taking all and only cost-justified risk reducing precautions, (b) imposing the responsibility of determining which risks are worth reducing and at what cost on those in the best position to make those decisions and to implement cost-justified precautions, and (c) distributing risks that are too costly to prevent in ways that reduce their impact on individual persons and time frames. 

(2) A successful organization implements plans to identify talent, access, develop and project it, and does so while managing risk in accordance with the principles stated above.

 (3) The most consequential and difficult judgments a team makes are those projecting how talent is likely to perform at the next level up, and especially at the major league level.  

(4) Normally, judgments of whether talent projects to performance at the major league level comes late in the development phase which implies that considerable resources have already been spent on player development, and the evidence on which the judgment is made is presumably substantial.

(5) Nevertheless, in spite of the extent of the evidence available at the time projections must be made, the level of uncertainty surrounding those judgments in conjunction with the consequences for the organization of making poor judgments has led to organizations  

(6) Pursuing alternative strategies for constructing their major league rosters they believe are less risky and fraught than those that rely on uncertain judgments about their own prospects' expected performance, e.g. trades and free agency, even though:

    (a) shifting to trade and free agency strategies does not obviate the need to make precisely the sort of talent projections they are seeking to avoid,

    (b) lead to poor trades and inefficiently long contracts of 'proven' talent precisely because they are bad at making talent projections, and

    (c) to overpaying for free agents both in terms of compensation and length of contract, thereby

    (d) creating the most inefficient approach to dealing with the uncertainty surrounding talent projection, that, in addition, does nothing to avoid having to make talent projections that choosing the alternative strategies in the first place is designed to avoid. 

Whatever strategy for roster construction an organization ultimately adopts, there is no escaping the need to project talent into performance.  One might as well address the challenge directly.  

Meeting the challenge amounts to developing evidence-based predicates for projecting performance thereby reducing uncertainty, increasing confidence in one's judgment and increasing the likelihood of developing an optimal use of home-grown talent, trades and free agency to construct rosters capable of regularly competing for titles over a sustained period of time.

In this and the next post (tomorrow) I explore several steps an organization should take (steps, I argue, the Mets have taken), to develop an evidence-based confidence in projecting performance at the next level. 

What organization behavior reveals

In order to solve the problem, we must correctly identify it, and uncover its source. Let's begin with organizational behavior, which, like behavior in general, can be revealing of motivations.  The question is what does the behavior of taking on substantial costs through trades and free agency reveal about the sources of uncertainty that plague talent projection judgments? I focus on two plausible inferences we should draw, the first of which is obvious, though both are important. 

(1) Teams are typically more confident in projections based on major league performance than they are on judgments based on what is often extensive information drawn primarily from minor league (as well as college and high school and other forms of organized baseball) performance.

(2) Teams are especially comfortable relying on the information provided by markets, that is, when other teams are pursuing trades for players made available and when teams in situations like the one's they face are bidding on players in free agency. 

Why are these inferences interesting?  

(1) They reveal that all teams believe that the best evidence of performance is prior performance, but more importantly that

(2) Performance at the major league level is generally significantly more trustworthy than performance at all levels of play inferior to it.  (More nuance is required to determine the extent of relative trustworthiness, etc.)

(3) Because talent projection is so riddled with uncertainty, teams are willing to enter the trade and free agent markets and thereby pay significantly more for 'proven' talent not just because the talent is proven, but also because they interpret other teams entering the bidding in those markets as factors strengthening the rationality of their bidding for those players as well, thus enabling them to feel better about additional costs they incur by choosing to construct rosters around trades and free agency. 

If this seems a little crazy to you, you won't get an argument from me. 

After all, the argument appears to have something like the following internal logic.  I am not confident in my ability to project my prospects' likely performance at the major league level.  So I will try to avoid having to rely on judgments I have inadequate confidence in -- to the extent possible, given my budget.  So far, so good.

I believe I can avoid having to rely on judgments I have inadequate confidence in by relying instead on making trades and dipping into the free agent markets.  Of course I realize that I can't fill my entire roster this way, so I know I will have to make some judgments about my own players.  But I want to minimize those and if possible only make judgments that I am most confident about. Again, my budget will impact the extent to which I can rely on trades and free agency to compensate for my lack of confidence. 

In entering markets in which I have to bid against others to secure 'proven' talent, I have made myself vulnerable to taking on higher costs to obtain talent.  The more bidders in the competition, the higher the price tag of winning.  The price is worth it to me.  I am bidding to secure proven talent.  I am avoiding the consequences of making a mistaken judgment in projecting my player's performance at the next level.  And I am purchasing confidence in the strategy I have adopted.  

So rather than lamenting the fact that others are involved in the bidding, which drives the price I have to pay up, I'm happy to pay the premium because the fact that they are in the market for the talent in question and bidding for the player's service itself confirms the rationality of my being in the market and bidding along with them, and therefore my confidence in what I am doing. If my bid wins, I am pleased with having adopted this strategy.  

There are many flaws in this argument, but one stands out above the rest. If I am driven to bid on players in trade and free agent markets by my lack of confidence in my ability to project performance from talent, what reason do I have for thinking that this isn't what also drives the behavior of my competitors who are also bidding in the trade and free agent markets.  

It's not just that they are in the same markets I am in for the same lack of confidence in player projections that we all share.  Their confidence that they are right to stay in and bid up the price is largely a function of their interpreting the market signals exactly as I am.  In other words, the presence of other bidders bidding grounds my confidence applies to them as well.  Everyone is paying a premium to buy confidence that consists entirely in doing what everyone else is doing on the assumption that the other bidders have information about the proven talent's projected performance that you don't have.  But if everyone is acting initially and then thereafter for the same or similar reasons, confidence is being manufactured out of thin air.  It need have no basis in factual information at all. 

This is a genuine 'confidence game', a veritable Ponzi scheme for creating confidence from the thinnest of air.  A great strategy to adopt if you are on the right side of a stock 'pump and dump' scheme, but not a promising strategy around which to organize roster construction in baseball. 

Where you look for talent matters

It makes more sense to see if you can develop a basis for genuine confidence in judgments about your own players likely performance at the next level.  The question is, where to begin.

If you want to be more confident in projecting performance at the highest level, start the process with the best talent you can find.  Start with A or A+ quality prospects, and not only will you have confidence about your projections, but your floor is so high that with appropriate development you've put yourself in a position to be reasonably confident in your projections without very much additional investment in addressing the uncertainty that remains. 

Makes sense?  No doubt.  How likely is this approach to work.  Well, if players could reliably identify themselves as highly likely to be major league level performers while still in their teens, then the strategy would be simple and work for everybody.  Who wouldn't want to start the process with nothing but A and A+ level players.  

Unfortunately, young players do not wear the range of their future performance on their sleeves, thus making them harder to identify -- to say the least. We are left to make those judgments ourselves and to gather evidence sufficient to do so. It would be great to start from a place where the floor of projected performance is just inches away from the highest of high ceilings.  Short of reliable self-identification all we would have accomplished is shift the most difficult decisions about talent projection (which now takes the form of determining who is an A or A+player) to an earlier stage in a player development when there is precious little information available on which to make any such judgment!

Still, there is something to be said on behalf of reassessing where an organization goes shopping for talent at the outset of the process, especially if one can sort among potential talent pools and identify ones that are more likely to produce major league level talent.  This is precisely the strategy college basketball coaches employ when developing relationships with high school and AAU coaches in states known to have high level high school basketball programs, e.g Florida, Texas and California, and maybe New York.  (So sorry but NYC high school basketball isn't what it once was.) 

With that in mind, consider the following:

* Roughly 5% of international free agents make it to the majors, but

* Between 25-30% of major league rosters are comprised of international free agents.

* Of those, the vast majority come from Venezuela, Cuba and the Dominican Republic.

* The majority of those signed to organization contracts are between 16-18 years old.

What can we glean from the data?  

(1) Focusing on smaller, easy to identify geographic regions in which players are locally sorted and identified as having some non-trivial baseline level of talent reduces search costs. 

(2) Leagues and development programs for sorting that talent exist and are generally well-functioning. This leads to early sorting into talent tiers which is helpful in reducing overall developmental costs.

(3) Additional marginal investments by major league clubs (including especially those who want to shop heavily in this pool, e.g. the Mets) will increase both the effectiveness of local assessment and development programs while strengthening relationships with local scouts.

(4) Early sorting not only reduces development costs, but allows for more targeted expenditures, likely to lead to better skill and talent development even before players are available to sign with major league organizations.

(5) Most importantly, given that nearly 30% of players on current major league squads come from this pool, the investments in search and development produce benefits that swamp the costs the investments require.

If you want to increase confidence in talent projection, search in the places where high end talent is abundant; invest early in identifying, sorting and developing it.  Investments strengthens all important relationships and improves initial assessments, reduces development costs and allows for targeted spending in development that increases confidence in projecting performance ranges. 

If recent trends in IFA signings are any indication, there is no question but that the Mets have learned this lesson well. If IFA market is the place to shop, the Mets have made it clear that they are all in.

Ok. we know where to start, but where do we go from there?

Upon taking the Indiana job, coach Curt Cignetti had to dig relatively deeply into the portal along with taking more than a few of his players from James Madison University. When asked what he was looking for in dipping into the player portal, he offered a one word answer, 'performance.' 

Cignetti was not disparaging skillsets as a basis for projecting performance so much as expressing a preference for skillsets that have been realized in actual performance.  And not just any old performance either, but performance at a level of competition that exceeds, matches or approximates the level at which Indiana would be competing. 

Both points are important and provide keys to improving a team's capacity to make reliable evidence-based projections about player performance at the major league level.

If the first key to improving talent projections is to look for talent in the right place, the second is to base one's projections on actual performance, and not directly on skillsets or other components of talent. 

Why would performance be the most reliable predicate for projecting future performance?  The answer is that performance as such is not, but that performance under certain conditions is.  In the next post I identify what some of the most important of those conditions are and explain why their presence contributes to the reliability of performance projections.

I also argue that the Mets have set up their minor league system, intentionally, I'd like to think, in order to put several of these conditions in place, thereby putting themselves in an especially good place for implementing an optimal and sustainable approach to roster construction: one that will be central to achieving their goals of consistently competing at highest level and being rewarded with championships over a sustained period of time (hopefully one that overlaps with my remaining years).

Stay tuned.  See you tomorrow.