How to Set TTL for Redis Set’s members

Redis is a popular in-memory data store known for its versatility and speed. It offers a range of data structures, including Sets, which allow you to store unique elements.

While Redis provides a built-in mechanism to set an expiration time (TTL) for keys, it doesn’t have a direct feature to set TTL for individual members within a Set. However, with some creative data modeling and a few Redis commands, we can achieve the desired behavior. In this tutorial, we’ll explore an approach to set TTL for Redis Set’s members.

The Idea

To set TTL for individual members of a Set, we can leverage Redis Sorted Sets. Sorted Sets are similar to Sets but with an additional associated score for each element. Imagine you are storing a pair rather than a single element in the set. This score is used to order the elements in the set. By utilizing the scores, we can effectively set TTL for each member in the Set. We can query, and delete ranges based on score. For e.g., we can get/delete values with scores between 0 to 5.

We will use a sorted set to have TTL on members of a set. So the idea is that while adding an element to a set, we will not just add the value but will set a score to that element as well and the score will be the “Current Timestamp + desired TTL”. For e.g., if we want to keep TTL of 1 day, we will set the score to be Date.now() + 1000*60*60*24.

Flow

Date-wise flow of data

• On the first day, we are adding some values (e.g., Val1, Val2, Val3) to the set and the score of each element will be Day 1 + 1 Day (in epoch format, i.e., Day 2).
• On day 2, when we rerun the code to add more elements, we will first add all the new values to our set (e.g., Val3, Val4, Val5). After this, our set will have (Val1-Val5) with (Val1, Val2) score equal to Day 2, and (Val3, Val4, Val5) score equal to Day 3.
• After adding the data, we will remove all the keys whose TTL is before Today i.e. Day 2. As Val1 and Val2 have scores set to Day 2, those values will be removed.
• When reading the values, we perform a range query from the current time to infinity, retrieving the values that are not expired.
• So when we read on Day 1 or Day 2, we will read the values that are not expired.

We will also remove the elements on first day as well but none will be removed in that. I’ve not kept it in the diagram to keep it simple.

Code

We will see the above implementation in JS using ioredis library to connect to Redis.

const Redis = require('ioredis');
const client = new Redis();
const key = 'test';

const startTime = Date.now();
const offset = 1000*60*60*24; // 1 day from now, configure as needed

async function save() {
    for (let i = 0; i < 10; i++) {
        await client.zadd(key, Date.now() + offset, i);
    }
    console.log('Saved');
    await client.zremrangebyscore(key, '-inf', startTime);
    console.log('Deleted old data');
};

save();

Let’s understand the code

1. Here we are first connecting to Redis using ioredis. You can connect any way you want.
2. Then we are taking an offset value which will be the TTL of the members.
3. After that, we have a function called save which adds the values in the sorted set using zadd method.
4. We are setting the score equal to Date.now() + offset saving it as expiry.
5. After saving all the values, we are deleting all the values before today i.e. the start time using zremremovebyscoremethod which deletes a range based on score. With this, we keep values only starting today.
6. These values will be deleted tomorrow when the script will run and hence keeping the TTL to 1 day(which can be configured).

Let’s see the code to read values

const Redis = require('ioredis');
const client = new Redis();

const key = 'test';
client.zrangebyscore(key, Date.now(), 'inf', (err, members) => {
    if (err) {
      console.error('Error retrieving sorted set members by score:', err);
    } else {
      console.log('Sorted set members within score range in reverse order:', members);
    }
});

In the above code

1. We are connecting to the Redis client using ioredis.
2. We use zrangebyscore method to get the elements between a range. Our range is between Current Time and infinity.
3. With this, we will only get the values that have TTL more than Date.now().
4. As while saving, we set the score to CurrentTime + 1 Day, we can use that data till 1 day covering our use case.

If your use case is to remove the data without depending upon the adding job or you’ll add the data only once. You may want to write a cronjob that will delete the old data occasionally as well. Though the get code will respect the score and will treat it as TTL but if you don’t want the sorted set to become big, you can use a cronjob. The idea is simple, you can do it any way you want.

Performance

When comparing Sets and Sorted Sets, it’s important to note the difference in performance:

• SMEMBERS: O(N), where N is the cardinality of the Set.
• ZRANGEBYSCORE: O(log(N) + M), where N is the number of elements and M is the number of elements fetched.
• SREM: O(N), where N is the number of elements being removed.
• SREMRANGEBYSCORE: O(log(N) + M), where N is the number of elements and M is the number of elements removed.
• SADD: O(1)
• ZADD: O(log(N)), where N is the number of elements in the Sorted Set.

While Sets generally provide faster performance due to their unordered nature, using Sorted Sets for TTL management introduces some overhead due to maintaining the order of elements. This overhead becomes more noticeable with large sets. However, for smaller sets, the performance difference should be negligible.

Conclusion

In this article, we’ve explored a technique to set TTL for individual members within a Redis Set using Redis Sorted Sets. By leveraging scores and specific Redis commands, we can effectively manage the expiration of Set members. We’ve provided code examples in JavaScript using the ioredis library to illustrate the implementation. Feel free to adapt and customize the code to fit your specific use case.

You can find the complete code example here.

By employing this approach, you can extend the TTL capability to individual members of a Set in Redis, providing more granular control over expiration and enabling advanced data management scenarios.