Redis Memory Optimization and Internal Encodings

Move from "Redis picks a compact encoding for small collections automatically" — the passive behavior the Core Data Types concept covers — to the three things Redis in Action Chapter 9, "Reducing memory use," shows you actively doing to cut memory further: keeping structures short enough to earn that automatic encoding, sharding one large logical structure across many small physical keys so each shard stays inside the compact-encoding thresholds, and packing bits and bytes directly into Strings for data that doesn't need a structure at all. The chapter's own numbers make the stakes concrete: "these methods helped me to reduce memory use from more than 70 gigabytes, split across three machines, down to under 3 gigabytes on a single machine." This concept covers the two of those three techniques that are genuinely distinct design decisions — sharding and bit/byte packing — rather than the encoding-threshold mechanics already covered in Redis Core Data Types: Strings, Lists, and Hashes.

• A single lookup table that has grown too large for one key — the book's own example: a HASH mapping 370,000+ city IDs to city info, sharded into many smaller HASHes so each shard stays under the listpack thresholds.
• Counting unique events at a scale where a single SET would be too large — a sharded SET of truncated visitor IDs to keep a live daily unique-visitor count without one giant SET.
• Storing a small, fixed-size record per sequential ID for a huge population — per-user location codes (country/state) packed 2 bytes at a time into sharded STRINGs, for a user base larger than any single String can hold.
• Caching millions of small string values as a plain key-value store — the same "split the key, use the tail as a hash field" pattern Redis's own memory-optimization documentation still recommends today for exactly this case.
• Dense per-ID boolean or counter flags — SETBIT/GETBIT/GETRANGE/SETRANGE on a String, the same primitive the Advanced Data Types concept's Bitmaps section builds on, used here for arbitrary fixed-width fields rather than one-bit flags.

Why short structures matter here, briefly

The mechanism itself — ziplist-style compact encoding below a configurable size threshold, promoted to a full structure above it — is Redis Core Data Types's territory, including the Redis 7.0 ziplist-to-listpack rename. What's worth restating here is why it saves so much: a doubly linked LIST node needs "three pointers, two integers... plus the string and an extra byte," which the book measures at "21 bytes of overhead to store 3 actual bytes of data" for a short string — versus a ziplist's "roughly 2 bytes" of overhead for the same entry. That gap is the entire reason sharding is worth doing: keeping each shard's structure inside the compact-encoding limits is what makes the memory savings below actually materialize, and Redis in Action is direct about the price of missing that limit — pushing a ziplist-encoded LIST from 1,000 to 100,000 entries dropped RPOPLPUSH throughput from roughly 50,000 ops/sec to 553 ops/sec, because "ziplists are effectively unusable" once they grow that large. Sharding is, in effect, a way to keep enjoying the short-structure discount at a scale that would otherwise force every individual structure past its limit.

Sharded structures: splitting one logical structure into many physical keys

The core idea, in the book's own words: "instead of storing value X in key Y, we'll store X in key Y:<shardid>." A shard_key() function decides which physical key a given logical entry belongs to: for sequential numeric keys, shard_id = key // shard_size (numerically adjacent keys land in the same shard); for non-numeric keys, a CRC32 checksum of the key modulo a computed shard count spreads entries roughly evenly — "we're using CRC32 in this case because it returns a simple integer without additional work, is fast to calculate... and because it'll work well enough for most situations." Both total_elements (expected total entries) and shard_size (target entries per shard) feed that calculation, and the book flags a real operational constraint: "you shouldn't change either of these values, or when you do change them, you should have a process for moving your data from the old data shards to the new data shards (this is generally known as resharding)."

Sharded HASH. The book's worked example shards the 370,000-entry city-lookup HASH by numeric city ID. Every HSET/HGET becomes shard_hset/shard_hget, which compute the shard key and then delegate to the ordinary HASH command on that shard. The payoff: "On a 64-bit machine, storing the single HASH of all of our cities takes up roughly 44 megabytes. But with these few small changes to shard our data... the sharded HASHes together take up roughly 12 megabytes. That's a 70% reduction in data size, which would allow us to store 3.5 times as much data as before." The book generalizes this beyond one big HASH: "If you find yourself storing a lot of relatively short strings or numbers as plain STRING values with consistently named keys like namespace:id, you can store those values in sharded HASHes for significant memory reduction in some cases" — the same per-key-overhead argument the Core Data Types concept's Instagram case study makes (21 GB of per-media-ID String keys down to roughly 5 GB as Hashes), just applied one level further by also splitting the Hash itself into shards.

Sharded SET. For counting unique daily visitors, storing full 128-bit UUIDs as SET members would be both large and unable to use the compact intset encoding. The book instead truncates each UUID to its first 56 bits (kept as an integer, so set-max-intset-entries still applies), justified by a birthday-collision calculation: "as long as we have fewer than 250 million unique visitors in a given time period... we'll have at most a 1% chance of a single match," and "if we have fewer than 25 million unique visitors, then the chance of not counting a user falls to the point where we'd need to run the site for roughly 2,739 years before we'd miss counting a single user." Sharding this truncated-ID SET (via the same shard_key() function, prefixed with a non-numeric character since the IDs aren't densely packed) and tracking an expected-size estimate that grows with yesterday's actual count produces a concrete number: "Redis will use approximately 9.5 megabytes to store the unique visitor count. Without sharding, Redis would use 56 megabytes to store the same data... That's an 83% reduction in storage with sharding, which would let us store 5.75 times as much data with the same hardware."

What the book deliberately does not shard. LISTs, because sharding one without Lua scripting is "difficult" — the book defers that to a later chapter's Lua-based implementation. ZSETs, because commands like ZRANGE, ZRANGEBYSCORE, ZRANK, and ZCOUNT "require operating on all of the shards of a ZSET to calculate their final result," which "violate[s] almost all of the expectations about how quickly a ZSET should perform" — sharding would defeat the entire reason to use a ZSET. The book's fallback for a genuinely huge ZSET is narrower: keep an auxiliary top/bottom-scoring ZSET trimmed with ZADD/ZREMRANGEBYRANK, answering "who's in the top N" without ever materializing the full sharded structure's range queries.

Packing bits and bytes into Strings

Sharding still stores each logical value as a first-class SET/HASH member. Packing goes one step further: store fixed-width fields as raw offsets inside a String, using GETRANGE/SETRANGE (byte-range read/write) and GETBIT/SETBIT (single-bit read/write) — "with these four commands, we can use Redis STRINGs to store counters, fixed-length strings, Booleans, and more in as compact a format as is possible without compression."

The book's worked example packs per-user location data — country plus, where available, US/Canada state or province — into exactly 2 bytes per user: one byte as an offset into a sorted COUNTRIES table, one byte as an offset into that country's STATES table (0 meaning "not found," so uninitialized data reads correctly as "unknown" rather than a valid index). Because Redis STRINGs are capped at 512 MB and a population of "750 million users today... would need over 1.5 gigabytes of space," the book shards the packed data across many STRINGs too — USERS_PER_SHARD = 2**20 (about 1,048,576 users, "just over 1 million entries") per shard, each shard landing at "about 2 megabytes per STRING." SETRANGE writes each user's 2-byte code at offset = (user_id % USERS_PER_SHARD) * 2 in the shard location:<shard_id>; a companion ZSET tracks the highest user ID seen so far, so an aggregation pass knows how many shards actually need scanning. Reading back proceeds in blocks — pull chunks of a shard's raw bytes, decode every 2-byte pair back into a country/state pair via the same lookup tables, and accumulate counts — which is how the book computes country/state distributions over either the entire population or an arbitrary provided list of user IDs (follower-location analytics, for instance), batching lookups into pipelines rather than one round trip per user.

The book is explicit that this is the natural extension of SETBIT/GETBIT, not a different technique: "we stored multiple bytes of data per user, [but] we can use GETBIT and SETBIT identically to store individual bits, or even groups of bits" — the same String-as-fixed-width-array idea the Advanced Data Types concept's Bitmap section uses for one-bit-per-user flags scales up cleanly to 2-byte, 4-byte, or arbitrary-width records with GETRANGE/SETRANGE.

Book vs today

The ziplist-to-listpack rename that both this technique and its underlying encoding depend on is covered in Redis Core Data Types: Strings, Lists, and Hashes — not repeated here.

The sharded-HASH pattern is still official Redis guidance today, not a book-era workaround. Redis's current memory-optimization documentation dedicates a full section, "Using hashes to abstract a very memory-efficient plain key-value store on top of Redis," to essentially the same idea: split a key like object:1234 into a HASH key (object:12) and a field name (34) so that "every hash will end up containing 100 fields, which is an optimal compromise between CPU and memory saved." Its own benchmark against 100,000 plain STRING keys: "USE_OPTIMIZATION set to true: 1.7 MB of used memory. USE_OPTIMIZATION set to false: 11 MB of used memory" — over 6x smaller, the same order of magnitude the book's HASH-sharding example reports. Curiously, that same current doc page still carries a config-directive name from its original Redis 2.2-era write-up, hash-max-zipmap-entries — a name from before ziplist existed, one rename older than the ziplist-to-listpack change; on current Redis the equivalent knob is hash-max-listpack-entries. So the technique is current, but that particular corner of the docs preserves two renames' worth of stale terminology in one warning box.

The sharded-SET pattern for approximate unique counting has been substantially superseded, though — by a tool the book didn't have. HyperLogLog (covered in Advanced Data Types) answers exactly the question the book's sharded-SET example solves — "roughly how many distinct visitors today" — in a fixed ~12 KB regardless of scale, against the book's own 9.5 MB sharded result (and 56 MB unsharded) for a comparable visitor count. The trade HyperLogLog makes that the book's sharded SET does not is giving up exact counts and membership queries; when that trade is acceptable — and for "how many unique visitors" it usually is — HyperLogLog is both simpler to implement and far smaller than manually sharding a SET. The sharded-SET technique remains correct and relevant for cases that need exact counts or membership answers (sharded SETs still support SISMEMBER on a specific shard, HyperLogLog fundamentally cannot); it's the "just get me an approximate count" case that today has a purpose-built, smaller answer the book's chapter predates.

Redis's larger default thresholds since the book's era reduce, but don't eliminate, how often manual sharding is needed. hash-max-listpack-entries defaults to 512 today (Redis 7+) — the same default the book's hash-max-ziplist-entries used — so a Hash needs the same few hundred entries before it needs sharding either way; nothing there has loosened. What has changed is that Redis's compact encodings now cover more of the type surface (Sets got set-max-listpack-entries in Redis 7.2, on top of the book-era intset), so a wider range of small collections get the automatic discount without any application-level sharding at all. Manual sharding is still the documented, correct move once a single logical collection is expected to exceed those thresholds by orders of magnitude — it just now needs to be reached for less often than in 2013.

• Sharding trades a single structure's full command surface for a subset of it, on purpose. The book shards HASHes and SETs but explicitly declines to shard ZSETs or LISTs without extra machinery, because sharding only pays off for the commands that can be answered by touching one shard (HGET, SADD, SISMEMBER) — commands that inherently need the whole structure (ZRANGE, ZRANK, ordered LIST access) either require fanning out to every shard, defeating their original performance guarantee, or need to be implemented as a genuinely different algorithm (Lua-scripted sharded LISTs) rather than a drop-in replacement.
• total_elements and shard_size are a commitment, not a tunable. Changing either after data has been written changes which shard every key maps to, silently splitting a logical collection across old and new shard assignments unless a real resharding migration moves the data. This is a heavier operational cost than adjusting hash-max-listpack-entries, which only affects internal encoding, not which key a value lives in.
• Bit/byte-packed Strings are the most memory-efficient option and the most brittle one. There is no field name, no type tag, and no self-description in a packed String — the COUNTRIES/STATES offset tables the book keeps in application code are the only thing that makes a 2-byte record meaningful. A version mismatch between the code that wrote the data and the code that reads it back (an updated country list with different offsets, for instance) silently decodes every record to the wrong location with no error raised.
• A single large String is capped at 512 MB, and growing one near that limit is itself expensive. The book notes that "due to Redis's clearing out of data when setting a value beyond the end of an existing STRING, setting the first value at the end of a long STRING will take more time than would be expected for a simple SETBIT call" — another reason packed data gets sharded across many STRINGs well before it would hit the 512 MB ceiling, not just to stay under it.
• Short structures, sharding, and packing compose, but each optimization narrows what's still easy to change later. Once city data lives in sharded HASHes with a fixed shard count, or location data lives in packed 2-byte codes across sharded STRINGs, adding a new field or changing the encoding format means touching every shard's data, not editing one key's value — the same kind of migration cost the Core Data Types concept flags for switching a String-keyed object into a Hash, just one level deeper.