rcoredump is meant to be a toolbox for aggregating, indexing, and searching core dumps. Think ELK for core dumps.
rcoredump is composed of two main softwares:
rcoredumpdis the indexer. It store, analyze, and index the core dumps and the associated binaries, and must be installed on the destination server.
rcoredumpis the forwarder. It collect core dumps and the associated binaries, and must be installed on the source servers.
The project is in its infancy stage. While it can be used already for some usages, many things may change between now and the v1, and no feature is guaranteed to stay the same or even be present in future releases. Of course, we will try to stay compatible as much as possible, but if we can’t, you’ve been warned.
The simplest way to install rcoredump is to use the repository’s release page. The binaries are currently built for Linux only, in 32 and 64 bits architectures. New systems and architectures can be added as long as they are supported by Golang. Feel free to open an issue if you would like to see binaries for your platform.
Note The current list can be listed using the command
go tool dist list
with any go installation.
Installing from source require only a working Golang installation. Your system probably provides a Golang package, you can also read the Getting Started page on Golang’s website. It is recommended to use the very last version of Golang, but any module-enabled version should do.
Then, you can either run the following commands to get it using golang’s package manager:
go get github.com/elwinar/rcoredump/bin/rcoredumpd go get github.com/elwinar/rcoredump/bin/rcoredump
Or clone the repository and build it by hand:
git clone https://github.com/elwinar/rcoredump cd rcoredump go build ./bin/rcoredumpd go build ./bin/rcoredump
Both binaries can be configured using either the CLI or a configuration file
with an ini-like syntax (see rcoredump.conf.example
and rcoredumpd.conf.example). Both usage message
can be printed on screen using the
Note The flag syntax follow the Golang’s standard package convention, described here. The same convention is applied for the configuration files, minus the dashes before the flag names.
Usage of rcoredumpd: rcoredumpd [options] -bind string address to listen to (default "localhost:1105") -c.analyzer string gdb command to run to generate the stack trace for C coredumps (default "bt") -conf string configuration file to load (default "/etc/rcoredump/rcoredumpd.conf") -data-dir string directory to store server's data (default "/var/lib/rcoredumpd") -filelog string path of the file to log into ("-" for stdout) (default "-") -go.analyzer string delve command to run to generate the stack trace for Go coredumps (default "bt") -index-type string type of index to use (values: bleve) (default "bleve") -retention-duration duration duration to keep an indexed coredump (e.g: "168h"), 0 to disable -size-buckets string buckets report the coredump sizes for (default "1MB,10MB,100MB,1GB,10GB") -store-type string type of store to use (values: file) (default "file") -syslog output logs to syslog -version print the version of rcoredumpd
Usage of rcoredump: rcoredump [options] <executable path> <timestamp of dump> -conf string configuration file to load (default "/etc/rcoredump/rcoredump.conf") -dest string address of the destination host (default "http://localhost:1105") -filelog string path of the file to log into ("-" for stdout) (default "-") -metadata value list of metadata to send alongside the coredump (key=value, can be specified multiple times or separated by ';') -src string path of the coredump to send to the host ("-" for stdin) (default "-") -syslog output logs to syslog -version print the version of rcoredump
By default, the forwarder expect the core dump to be passed on the standard
input. You can use sysctl’s
kernel.core_pattern tunable (on linux) to have
the kernel invoke rcoredump everytime a dump is generated. For example:
kernel.core_pattern=|/path/to/rcoredump %E %t.
Note No space between the
| and the binary’s path. Also, no environment
variable, so no
PATH, you must use an absolute path here.
The forwarder can also be invoked by hand using the
-src flag and a file
path. This is mostly used for development and to test an installation.
By default, all logging is done on stdout using the logfmt format. For
convenience, both softwares also accept a
-syslog flag to log using syslog,
-filelog flag to log to a file.
Right now, the indexer stores every core dump and binary on disk. As both of
them can be quite large, it is recommended to closely monitor the size of the
data directory (set by the server’s
-dir flag), and the free space on the
If non-zero, the
-retention-duration flag of the server can be used to
automatically remove coredumps older than the value, eventually removing the
executable if it is not linked to another coredump.
Feel free to open an issue, or contact me by mail at
firstname.lastname@example.org. I won’t guarantee and instant solution, but I’m
always happy to help someone get things working.
The project is open to contribution, but no guidelines or requirements are set for now, as this is mostly a one-man project. Feel free to contact me if you want to help, and we will define this part together.