Debunking Treasure Hunter and Drop Rates in FFXI

There are so many forum posts and blog entries citing everyone's opinion, and superstitions of how drop rates work in FFXI and how they are affected by Treasure Hunter. Because FFXI content is powered server-side, it is a little difficult for us to analyze the underlying mechanics of this phenomenon. However, it isn't so difficult for us to do so in other titles by Square Enix, including those with a Treasure Hunter trait. Instead of analyzing other people's theories on this issue and picking them apart and reassembling them into my own theory based on the most plausible ideas, I decided to go into a different direction. I also believe similar mechanics drive the crafting system, and I'll probably be addressing that in the near future.
Let me begin by defining what I believe is wrong with the current theories. First of all, they all seem to define the drop rates in exact percentages and that Treasure Hunter bonuses apply a fixed percentage. I don't believe this works at all in the current model, and the program mechanics to drive this theory would be outlandishly and unnecessarily bulky. Also, I don't believe that weather and moon phases affect this process because that's just crazy talk that makes no sense. Furthermore, I don't believe there's a drop percentage higher than 50% other than 100% for any item set to a specific rarity as this is beyond the scope of game mechanics, however, I do believe similar items with different or similar rarity values can combine to make it seem like a drop percentage is between 50% and 100%. Finally, I believe there may be specific caps for the number of times the same item can drop off the same monster even though the same item may occupy more than one slot or a percentage may actually factor to something higher than 100% and wind up being forced to cap at 100%.
As far as Treasure Hunter, I believe it simply negates the rarity of eligible items and testing would be required to see how Treasure Hunter and Treasure Hunter II combine to affect this model. If broad testing were conducted with the following numbers, I believe we could accurately deduce and assign the base rarity of any item in the game and the affects that Treasure Hunter has on these items.
Analyzing Final Fantasy games that we can, I believe that the drop rates are defined in a base 2 system. Similar systems seem to govern the foundation of all underlying games that we understand and I don't believe Final Fantasy XI is exempt from these basic programming mechanics. While looking at the proven model for Treasure Hunter in other Final Fantasy titles, it seems to me that all it does is nudge the item down the scale, effectively doubling its drop rate. Since the system is base 2, the doubling effect distorts actual percentage values which are in a base 10 system. This distortion causes different people to experience different results, and encourages the superstitions that outside forces are affecting them.
We'll have to do some testing to define the highest possible rarity in FFXI, but I believe it is 1/256, or roughly .390625% (slightly less than half a percentage point). I believe the other base drop rate values are simply 1/128 (.78135%), 1/64, (1.5625%), 1/32 (3.125%), 1/16 (6.25%), 1/8 (12.5%), 1/4 (25%), and 1/2 (50%). Of course, there's also the 100% drop rate which I don't think needs to be mentioned. Using these numbers, I believe what "Treasure Hunter +1" does is simply add 1 to the first number in the ratio and effectively doubling the drop rate. However, because the doubling is done in a base 2 system, the affect of the percentage doubling is distorted. These numbers reveal this distortion for what it really is.
Another thing this model explains is why drop rates seem to be either pretty good or pretty terrible. At the lower end of the scale, doubling a terrible drop rate still results in a horrible percentage rate, however doing so at the higher end of the scale seems to dramatically improve the drop rate.
Let's look at this from Treasure Hunter's perspective now. If we have a drop rate of 1/128 and Treasure Hunter adds +1 to the 1/128 we get 2/128. That number factored down is 1/64, which is the next drop rate tier. Does that mean having a Thief's Knife and Assassin's Kote would give us a drop rate of 3/128 or 1/32? It would be difficult because the difference between 2.34375% and 3.125% isn't even an entire percentage point and these numbers are low enough as it is. In order to decipher this we would have to go up the scale to and try to test the difference between something like 3/8 (37.5%) and 1/2 (50%). If we've established that a specific item has a 1/8 drop rate without the influence of Treasure Hunter, we could use "Treasure Hunter +2" to see if the addition is applied to the original ratio or simply scales the ratio down. The difference between these ratios is pretty steep at 12.5%, so I believe that we could deduce this even with smaller scale testing conducted by only a couple of players. I wouldn't be surprised if the actual ratio was 3/8 because this would make more sense to why so many of the drop rates seem confusing. The reason would obviously be the very high number of ratio possibilities because of the variables imposed by the dynamic intervention of Treasure Hunter between tests. Instead of having a model with an obviously definite base 2 system, the introduction of varying degrees of Treasure Hunter distorts the model in a realistic way.
Now, I would like to broaden the above model with the possibility that the wording between the two Treasure Hunter job traits available to Thief are mean to define a logical difference between their application. The above sort of hypothesizes that Treasure Hunter effectively doubles the drop rate, and that Treasure Hunter II would double the resulting drop rate. It also states that "+1" bonuses to Treasure Hunter affects the first number in the ratio which would cause a single Treasure Hunter bonus to double the drop rate defined after both Treasure Hunter job traits were applied. A second Treasure Hunter bonus causes a distortion in the entire model by given a ratio that doesn't reduce the fraction by another half. This would make more sense to me as many of us seem to see little difference between "Treasure Hunter 3" and "Treasure Hunter 4". I believe this is due to the possibility that the difference in these two isn't the result of a "doubling" effect.
I do want to stress the possibility that how Treasure Hunter II is applied could be similar to the Treasure Hunter bonus applications and that it's "Treasure Hunter 3" that actually gets the next doubling effect. What I mean by this is that Treasure Hunt itself may bump a drop of 1/64 to 2/64 which is factored down to 1/32. However, Treasure Hunter II, may just take a 1/64 drop rate and only bump it to 3/64 (4.6875%) instead of 1/16 (6.25%). That would mean that "Treasure Hunter 3" would be responsible for only lowering this drop rate to 4/64 or 1/16. If this were the case, the introduction of "Treasure Hunter 4" adds even more to the dynamics of this model by introducing other possible ratios such as 5/64 which only bumps the drop rate to a little over 8.3% rather than 3/16 (18.75)%
All this may seem pretty daunting, but I don't believe it would be that difficult for us to find an item's base rarity. Simply killing a couple hundred of monsters without Treasure Hunter could accurately define the more common items and the rare ones could be defined over the course of time. Also, it would allow us to insert accurate values to an item's rarity in the FFXI wiki and allow people to really gauge their need for Treasure Hunter for any specific item they need to obtain.
Due to my personal understanding of how drop rates are programmed as well as my personal observations of Treasure Hunter, this is my general theory of Treasure Hunter in FFXI and how it applies to the predefined rarity values of an item's drop rate. It is also the theory I'll stick to until someone disproves it. I would welcome any support I could get from those experienced in programming as well as general observation by members of the FFXI community regarding the application of this theory.
My final objective, with the aid of broad testing by other players, will be to narrow down the possibilities I have outlined in regards to the application of Treasure Hunter job traits and the bonuses granted by Thief's Knife and Assassin's Kote so that we have a definite system for calculating the drop rate of a given item. Once we become to understand the truth behind the system governing drop rates, we should be able to apply it to other things such as defining the exact reduction in drop rate imposed by "AoE farming" lower level monsters. I believe that its possible that such a method of farming has likely reduced common drops by as many as two tiers in this scale. Finally, I believe a similar system works for the crafting system which I've yet to outline as I'm still gathering data on the percentages of varying crafting tiers.
I'd like to summarize by recapping the varying possibilities of this model. I believe that any single item that can drop off a monster has a root value defining its rarity in a base 2 system. These values are 1/256(390625%), 1/128 (.78135%), 1/64, (1.5625%), 1/32 (3.125%), 1/16 (6.25%), 1/8 (12.5%), 1/4 (25%), 1/2 (50%) and 1/1 (100%). I believe that the Treasure Hunter job trait simply drops the rarity by a single tier effectively doubling the rate at which an item can drop. However, I believe that Treasure Hunter II may be more complex as it may simply add a bonus along with the first Treasure Hunter and create a ratio that simply can't be factored such as 3/256 (1.171875%) which would make more sense as rare items still seem to be rare even after several applications of Treasure Hunter. However, there's also the possibility that the bonus is applied after the initial Treasure Hunter and that the rate drop rate is effectively doubled again. This could also make sense because when applied to the rarest possible drop rate of 256, that would give it a drop rate of 1/64 (1.5625%) as opposed to 3/256 (1.171875%). The rarest possible items still remain pretty damn uncommon under the influence of either version of this formula. Treasure Hunter bonuses applied by the Thief's Knife and Assassin's Kote seem to point to the idea that original ratio itself is simply altered and result that can be factored is just a coincidence of the mechanics rather than a law governing the formula itself. That would mean that with "Treasure Hunter +3" your drop rate would be 4/256 which could be factored down to 1/64 and results in a percentage of 1.5625% when applied to the rarest possible drop rates. "Treasure Hunter +4" would likely reduce the rarity to 5/256 or 1.953125% making rare items still very rare even with with the highest possible rate of Treasure Hunter. Considering how some drop rates still seem to hover around 1% or 2% even with "Treasure Hunter 4" it is likely that this is the correct formula. However, even under the most optimistic circumstances that Treasure Hunter II reduced the rarity of a 1/256 all the way down to 1/64 and the bonuses from Thief's Knife and Assassin's Kote are applied afterwards, that still only gives us a maximum bonus to the rarest items of (3/64) or drop rate of 4.6875% compared to the 5/256 ratio that grants us a 1.953125% drop rate. Such small differences would be difficult to test but if we applied either formula to a base rarity of 1/32 we'd have a difference of 3/8 (37.5%) or 5/32 (15.625%). Finally, we would have to figure if multiple items are in the drop pool, if there's a cap on how many identical items can drop, and we have to find out if there are limitations that prevent fractions such as a drop rate of 1/4 being boosted to 5/4 or if that there are no limitations if this allows for an automatic drop rate or if the drop rate is rolled over into a new possible drop rate of an identical item. Of course, with an item of 100% drop rate that doesn't mean we'll get five them as I'm sure if this were the case there would be a cap imposed. Otherwise, items that drop 100% would have a drop ratio of 256/256 (1/1) and could be boosted to 260/256 or 5/1, depending on which formula for bonuses defines the model, and with no cap imposed we would be getting 5 of them under the latter formula and that just doesn't happen.
It has been over six years since FFXI was released and it is about time we define these fundamental mechanics and end all these stupid superstitions. Any assistance would be much appreciated, and hopefully, we'll have a working model to publish on the wiki by the end of the year.