Loot drop ratio and experience from slaying creatures drops when the difference of levels between the player and the creature is high enough.
For loot, it starts after the player is some levels higher than the creature (exact level difference varies according to player level range), and it drops until the ratio is 50% of the full ratio. For experience, it starts after the player is five levels higher and drops to zero.
The formula for the multiplicative modifier for experience or loot ratio is as follows:
modifier = 1 - 0.05 * (playerLevel - monsterLevel - (5 + FLOOR(playerLevel / 10)))
Calculated modifier is capped between 0.5 and 1
modifier = 1 / 1.2 ^ (playerLevel - monsterLevel - 5)