Redis is an in-memory database that persists on disk. The … – GitHub

This README file is just a quick and quick start document. More detailed documentation can be found at redis.io.

What is Redis?

Redis is often referred to as a data structure server. What this means is that Redis provides access to mutable data structures through a set of commands, which are sent using a server-client model with TCP sockets and a simple protocol. Therefore, different processes can query and modify the same data structures in a shared way.

The data structures implemented in Redis have some special properties:

  • Redis cares about storing them on disk, even if they are always served and modified in server memory. This means that Redis is fast, but it is also non-volatile.
  • The implementation of data structures emphasizes memory efficiency,

  • so data structures within Redis will likely use less memory compared to the same data structure modeled with a high-level programming language. Redis
  • offers a number of features that are natural to find in a database, such as replication, adjustable levels of durability, clustering, and high availability.

Another good example is to think of Redis as a more complex version of memcached, where the operations are not just SET and GET, but operations that work with complex data types like lists, sets, ordered data structures, etc

.

If you want to know more, here is a list of selected starting points:

  • Introduction to Redis data types. https://redis.io/topics/data-types-intro
  • Try Redis

  • directly within your browser. https://try.redis.io
  • The complete list of Redis commands. https://redis.io/commands
  • There is much more within the official Redis documentation. https://redis.io/documentation

Building Redis

Redis

it can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD. We support big endian and little endian architectures, and 32-bit and 64-bit systems.

It can be compiled on Solaris-derived systems (e.g., SmartOS), but our support for this platform is best effort and Redis is not guaranteed to work as well as on Linux, OSX, and *BSD.

It’s as simple

as: To build with TLS support, you will need OpenSSL development libraries (e.g. libssl-dev on Debian/Ubuntu) and run: To

build with systemd support, you will need systemd development libraries (such as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run

:

To add a suffix to Redis program names, use

: You can create a 32-bit Redis binary using:

After compiling Redis, it’s a good idea to test it using:

If TLS

is compiled, run the tests with TLS enabled (you will need tcl-tls installed):

Troubleshoot build issues with dependencies

or cached build options

Redis has some dependencies that are included in the deps directory. make does not automatically rebuild dependencies even if something in the source code of dependencies changes.

When updating the source code with git pull or when the code within the dependency tree is modified in any other way, be sure to use the following command to actually clean everything up and rebuild from scratch

:

This will clean: jemalloc, lua, hiredis, linenoise and other dependencies.

Also, if you force certain build options as the 32-bit target, without C compiler optimizations (for debugging purposes) and other similar compile-time options, those options are cached indefinitely until you issue a make distclean command.

Troubleshoot 32-bit

binary creation issues

If after compiling Redis with a 32-bit target you

need to rebuild it with a 64-bit target, or vice versa, you must perform a make distclean on the root directory of the Redis distribution.

In case of compilation errors when trying to build a Redis 32-bit binary,

Try the following steps: Install the

  • libc6-dev-i386 package (also try g++-multilib).
  • Try using the following command line instead of make 32bit: make CFLAGS=”-m32 -march=native” LDFLAGS=”-m32″

Allocator

Selecting a non-default memory allocator when compiling Redis is done by setting the MALLOC environment variable. Redis is compiled and linked against libc malloc by default, with the exception of jemalloc being the default on Linux systems. This default was chosen because jemalloc has been shown to have fewer fragmentation problems than libc malloc.

To

force compilation against libc malloc

, use:

To compile against jemalloc on Mac OS X systems, use:

Monodonic

clock

By default, Redis will be compiled using the POSIX clock_gettime function as a monotone clock source. In most modern systems, the processor’s internal clock can be used to improve performance. Precautions can be found here: http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

To compile with support for the processor’s internal instruction clock, use:

Detailed compilation

Redis will compile with easy-to-use colored output by default. If you want to see a more detailed result, use the following

: Running Redis To run Redis with the default settings,

simply type:

If you want to provide your redis.conf, you must run it using an additional parameter (the path of the configuration file):

It is possible to modify

Redis

settings

by passing parameters directly as options using the command line. Examples

:

All redis.conf options are also supported as command-line options, with exactly the same name.

Running Redis with TLS:

See the TLS.md file for more information on using

Redis with

TLS

. Play with

Redis

You can use redis-cli to play with Redis. Start a redis-server instance, and then, in another terminal, try the following

: You can

find the list of all available commands in https://redis.io/commands

. Installing Redis To install Redis binaries in /

usr/local/bin, simply use:

You can use make PREFIX=/some/other/directory install if you want to use a different target

.

Make Install will only install binaries on your system, but will not configure startup scripts and configuration files in the appropriate place. This isn’t necessary if you just want to play around with Redis a bit, but if you’re installing it the right way for a production system, we have a script that does this for Ubuntu and Debian systems

:

Note: install_server.sh won’t work on Mac OSX; it’s built for Linux only

.

The script will ask you a few questions and set up everything you need to run Redis properly as a background. Daemon that will start again when the system restarts.

You’ll be able to stop and start Redis using the script called /etc/init.d/redis_<portnumber>, for example /etc/init.d/redis_6379.

Code Contributions

Note: By contributing code to the Redis

project in any form, including submitting a pull request via Github, a code snippet or patch via private email or public discussion groups, you agree to release your code under the terms of the BSD license that can be found in the COPYING file included in the Redis source code distribution.

See the file CONTRIBUTING.md in this source distribution for more information. For bugs and security vulnerabilities, see SECURITY.md.

If you’re

reading this README, chances are you’re in front of a Github page or you’ve just broken the Redis distribution tar ball. In both cases, it is basically one step away from the source code, so here we explain the design of the Redis source code, what is in each file as a general idea, the most important functions and structures within the Redis server, etc. We keep the whole discussion at a high level without delving into the details, as this document would be huge otherwise and our codebase is continually changing, but a general idea should be a good starting point to understand more. In addition, most of the code is heavily commented and easy to follow.

Source code design

The Redis root directory

contains only this README, the Makefile that calls the actual Makefile inside the src directory, and an example configuration for Redis and Sentinel. You can find some shell scripts that are used to run the Redis, Redis Cluster, and Redis Sentinel unit tests, which are deployed within the test directory.

Within the

root are the following important directories

: src: contains the

  • Redis implementation
  • , written in C. tests

  • : contains the unit tests, implemented in Tcl
  • .

  • deps: contains libraries used by Redis. Everything needed to compile Redis is inside this directory; your system only needs to provide libc, a POSIX-compliant interface, and a C compiler. Notably deps contains a copy of jemalloc, which is the default Redis allocator under Linux. Note that in deps there are also things that started with the Redis project, but for which the main repository is not redis/redis.

There are a few more directories, but they are not very important to our goals here. We will focus mainly on src, where the Redis implementation is contained, exploring what is inside each file. The order in which the files are exposed is the logical one to follow to reveal different layers of complexity incrementally.

Note: lately Redis was refactored quite a bit. Function names and file names have been changed, so this documentation may more closely reflect the unstable branch. For example, in Redis 3.0 the files server.c and server.h were named redis.c and redis.h. However, the general structure is the same. Note that all new developments and pull requests must be made against the unstable branch.

server.h

The easiest way to understand how a program works is to understand the data structures it uses. So let’s start from the main Redis header file, which is server.h.

All server configuration and in general all shared state is defined in a global structure called server, of type struct redisServer. Some important fields in this structure are:

server.db is an array of Redis databases,

  • where data is stored
  • . server.commands is the

  • command table
  • . server.clients is a

  • linked list of clients connected to the server. server.master
  • is a special client, the

  • master, if the instance is a replica.

There are tons of other fields. Most fields are commented out directly within the structure definition.

Another important data structure of Redis is that defined by a customer. In the past it was called redisClient, now only client. The structure has many fields, here we will only show the main ones

: The client structure defines a

connected client:

  • The fd field is the descriptor of the client socket file
  • .

  • argc and argv are populated with the command that the client is executing, so that functions that implement a given Redis command can read the arguments.
  • querybuf accumulates client requests, which are parsed by the Redis server according to the Redis protocol and executed by calling implementations of the commands the client is executing.
  • Reply and BUF are dynamic, static buffers that accumulate the responses that the server sends to the client. These buffers are written incrementally to the socket as soon as the file descriptor can be written.

As you can see from the client structure above, the arguments of a command are described as robj structures. The following is the complete robj structure, which defines a Redis object: Basically

,

this structure can represent all basic Redis data types, such as strings, lists, sets, ordered sets, etc. The interesting thing is that it has a type field, so that it is possible to know what type a given object has, and a refcount, so that the same object can be referenced in multiple places without assigning it several times. Finally, the ptr field points to the actual representation of the object, which can vary even for the same type, depending on the encoding used.

Redis objects are widely used in the internal components of Redis, however, to avoid the overhead of indirect accesses, recently in many places we only used simple dynamic strings that are not wrapped inside a Redis object.

server.c

This is the

entry point of the Redis server, where the main() function is defined. The following are the most important steps to start the Redis server.

  • initServerConfig() configures the default values for the server structure
  • .

  • initServer() assigns the data structures needed to function, configure the listening socket, and so on
  • .

  • aeMain() starts the event loop that listens for new connections.

There are two special functions

periodically called by the event loop: serverCron() is

  1. called periodically (according to the server.hz frequency) and performs tasks that need to be performed from time to time, such as checking if clients have timed out.
  2. beforeSleep() is called each time the event loop is triggered, Redis serves some requests, and returns to the event loop.

Inside server.c you can find code that handles other vital things from the Redis server:

call() is

  • used to call a given command in the context of a given client
  • .

  • activeExpireCycle() handles key eviction with a set time-to-live through the EXPIRE command
  • .

  • performEvictions() is called when a new write command must be run but Redis has no memory according to the maxmemory directive.
  • The redisCommandTable global variable defines all Redis commands, specifying the name of the command, the function that implements the command, the number of required arguments, and other properties of each command.

commands.c

This file is automatically generated by utils/generate-command-code.py, the content is based on the JSON files in the src/commands folder. These are meant to be the single source of truth about Redis commands, and all metadata about them. These JSON files are not intended to be used by anyone directly, but the metadata can be obtained through the COMMAND command.

networking.c

This file defines all I/O functions with clients, masters, and replicas (which in Redis are only special clients):

  • createClient() assigns and initializes a new
  • client.

  • The addReply*() family of functions is used by command implementations to add data to the client structure, which will be transmitted to the client as a response to a given executed command.
  • writeToClient() transmits the pending data in the output buffers to the client and is called by the sendReplyToClient() writable event handler.
  • readQueryFromClient() is the readable event handler and accumulates data read from the client in the query buffer.
  • processInputBuffer() is the entry point for parsing the client query buffer according to the Redis protocol. Once the commands are ready to be processed, call processCommand() that is defined within server.c to actually execute the command.
  • freeClient() allocates, disconnects, and deletes a client.

aof.c and rdb.c

As you can guess from the names, these files implement RDB and AOF persistence for Redis. Redis uses a persistence model based on the fork() system call to create a process with the same (shared) memory content as the main Redis process. This secondary process dumps the contents of memory onto disk. This is used by rdb.c to create snapshots on disk and by aof.c to perform AOF rewrite when the append-only file becomes too large.

The implementation within aof.c has additional functions to implement an API that allows commands to add new commands to the AOF file as clients execute them.

The call() function defined within server.c is responsible for calling the functions that will in turn write the commands to the AOF.

db.c

Some Redis commands operate on specific data types; others are general. Examples of generic commands are DEL and EXPIRE. They operate in keys and not in their values specifically. All those generic commands are defined within db.c.

In addition, db.c implements an API to perform certain operations on the Redis dataset without directly accessing the internal data structures.

The most important functions within db.c that are used in many command implementations are as follows:

lookupKeyRead() and

  • lookupKeyWrite() are used to get a pointer to the value associated with a given key, or NULL if the key does not exist
  • . dbAdd() and its

  • top-level counterpart setKey() create a new key in a Redis database. dbDelete() deletes a key and its
  • associated value.
  • emptyDb() deletes a single entire database or all defined databases.

The rest of the file implements the generic commands exposed to the client

.

object.c

The robj structure that defines Redis objects was already described. Within object.c are all functions that operate with Redis objects at a basic level, such as functions for assigning new objects, handling reference counting, etc. The notable functions within this file:

incrRefCount()

  • and decrRefCount() are used to increment or decrease a reference count of objects. When it drops to 0, the object is finally released.
  • createObject() assigns a new object. There are also specialized functions for mapping string objects that have specific content, such as createStringObjectFromLongLong() and similar functions.

This

file also implements the OBJECT command

.

replication.c

This is one of the

most complex files within Redis, it is recommended to address it only after familiarizing yourself a bit with the rest of the codebase. This file contains the Redis master and replica role implementation.

One of the

most important functions within this file is replicationFeedSlaves() which writes commands to clients representing replica instances connected to our master, so that replicas can get the writes made by clients: this way your dataset will remain synchronized with that of the master.

This file also implements the SYNC and PSYNC commands that are used to perform the first synchronization between masters and replicas, or to continue replication after a disconnection.

Script

The script unit consists of 3 units:

script.c – script integration with Redis

  • (command execution, set replication/resp, …)
  • script_lua.c – responsible for executing Lua code, uses script.c to interact with Redis from within Lua code.

  • function_lua.c – contains the implementation of the Lua engine, uses script_lua.c to execute Lua code.
  • functions.c: contains the implementation of Redis functions (FUNCTION command), uses functions_lua.c if the function you want
  • to invoke needs the Lua engine.

  • eval.c – contains the eval implementation using script_lua.c to invoke Lua code.

Other C files

  • t_hash.c, t_list.c, t_set.c, t_string.c, t_zset.c, and t_stream.c contain the implementation of Redis data types. They implement an API for accessing a particular data type and client command implementations for these data types.
  • ae.c implements the Redis event loop, it is a self-contained library that is easy to read and understand.
  • sds.c is the Redis string library, see https://github.com/antirez/sds for more information.
  • anet.c is a library to use the POSIX network in a simpler way compared to the raw interface exposed by the kernel.
  • dict.c is an incrementally repeating implementation of a non-blocking hash table.
  • cluster.c deploys the Redis cluster. Probably a good read only after being very familiar with the rest of the Redis codebase. If you want to read cluster.c, be sure to read the Redis Cluster specification.

Anatomy

of a Redis command

All Redis commands are defined as follows:

The command function is referenced by a JSON file, along with its metadata, see commands.c described above for more details. Command flags are documented in the previous comment to struct redisCommand in server.h. For more information, see the COMMAND command. https://redis.io/commands/command/

After the

command operates in some way, it returns a response to the client, usually using addReply() or a similar function defined within networking.c.

There are tons of command implementations

within the Redis source code that can serve as examples of actual command implementations (e.g. pingCommand). Typing in some toy commands can be a good exercise in familiarizing yourself with the codebase.

There are also many other files that are not described here, but it is useless to cover everything. We just want to help you with the first steps. Eventually, it will find its way inside the Redis codebase 🙂 Enjoy it

!