cqueue is a C library that provides persistent, named data queues for programs. Arbitrarily many queues can co-exist, given they have distinct names.
Arbitrary binary data can be stored. Queues are manipulated using a stack-based API:
push()-- append data to the queuepop()-- pop data from the queuepeek()-- retrieve data from a given stack index
originally, I port libqueue from another github developer projects to use leveldb, but I found that queue_pop was extremely slow. I actually don't need or want the level's or key value aspects of level db so I wrote this as simple file based replacement.
I needed a simple durable fast queuing solution for one of my projects. I found a simple durable queue api at https://github.com/2ion/libqueue which was using Tokyo Cabinet for storage of which I didn't like the licensing. I ported it to use leveldb and found the performance for deleting (popping) records was extremely slow: [:~/libqueue] master+* ± ./bench it took 0 to push 10485760(inf) in 1024(inf) blocks it took 36 to pop 10485760(291271) in 1024(28.4444) blocks https://github.com/acgreek/libqueue so replaced leveldb with a simpl file based back-end
[:~/cqueue] master(+1/-1)* ± ./bench it took 0 to push 104857600(inf) in 1024(inf) blocks it took 1 to pop 104857600(1.04858e+08) in 1024(1024) blocks https://github.com/acgreek/cqueue note: I ran these tests on my hp chromebook running cruton linux
/*
Intransparent data type. Nothing to see here...
*/
struct Queue;
/*
Data structure used to store data in, and retrieve data from the
queue
*/
struct QueueData {
void *v;
size_t vlen;
};
/*
Functions return LIBQUEUE_SUCCESS on success, LIBQUEUE_FAILURE on
error or LIBQUEUE_MEM_ERROR as a special failure type for failed
memory allocations.
*/
/*
Open a named queue. Directory location is provided as string
<id>. Queues are persistent and data stored previously will still be
available through the name it was written to. Operations on a queue
must conclude with a call to queue_close().
This function ensures that ${path}/libqueue exist. If it doesn't
and cannot be created OR not written to, this function fails.
*/
struct Queue * queue_open(const char *path);
/*
Same as queue_open but allow one to provde options for various settings
The vararg must be terminated by a NULL to indicate end of options
options are to be provided as strings with some requiring args
available options:
maxBinLogSize byte size of the binlog, smaller binlogs cost performance in that as the queue grows and strinks, the more files will be created and destroyed. Larger binlog will use up more disk space for when the entries are deleted
maxEntries Max number of entries in the queue. queue_push will return LIBQUEUE_FAILURE if attempt is made to exceed this limit
maxSizeInByte Max number of appropriate disk usage of queue. queue_push will return LIBQUEUE_FAILURE if attempt is made to exceed this limit
*/
struct Queue * queue_open_with_options(const char *path,..., NULL);
/*
Push a data object onto the queue. struct QueueData must be filled
appropriately: d->v is a void* pointer to the data, d->vlen holds the
data size.
*/
int queue_push(struct Queue *q, struct QueueData *d);
/*
Pop data from the queue. The library will store the data in the passed
QueueData struct. d->v must be freed by the user.
*/
int queue_pop(struct Queue *q, struct QueueData *d);
/*
returns total size of all files used by the queue unless there are zero elements
in which it will return 0
*/
int queue_len(struct Queue *q, int64_t *len);
/*
returns numbers of elements in the queue
*/
int queue_count(struct Queue *q, int64_t *countp);
/*
compacts the queue on file system to use take up less disk space if possible
*/
int queue_compact(struct Queue *q);
/*
Like queue_pop(), but doesn't remove the data object from the queue
and returns the data at queue position s. Indexing is not used
*/
int queue_peek(struct Queue *q, int64_t s, struct QueueData *d);
/*
Like queue_push(), but over-writes the data at queue position s. This
function cannot be used to append new data to the queue. Indexing
starts from zero. s must be positive.
*/
int queue_poke(struct Queue *q, int64_t s, struct QueueData *d);
/*
Closes the queue. Only AFTER this function has been called one can be
sure that the data has been properly stored. Undefined behaviour may
occur if the concluding call to this function is omitted.
*/
int queue_close(struct Queue *q);struct Queue is an opaque data type the fields of which
should not be accessed directly.
Dependencies: none
libqueue uses cmake for building:
cmake . make
It's very simple: Use
#include <queue.h>in your C program and link it with
-lqueue given that libqueue was installed into the default linker search path.
The library comes with a number of reference/test/practical utilities that all operate on the queue named 'queueutils-stack0' to be used on the command line:
Pushes all its arguments in onto the queue queueutils-stack0. Exits with an error code of 0 if the push was successful and 1 if it wasn't.
Pops the strings from the queue that were pushed using
qpush, on element per call to qpush. If there is no element
to pop, exits with the status code 1. If the pop was successfu, it
prints the string to stdout and exists with the status code 0.
if there are no entries, then zero is returned or else the total size of all file used by the queue is returned
number of elments in the queue
not used
not implemented
Peek at elements in the queueutils' queue.
qpeek <INDEX> [<index1>[..<indexN>]]</indexN>]]Tries to peek at all indizes listed as arguments. Skips over invalid indizes but fails if the queue cannot be accessed properly.
Replace an existing in the queue with another one.
qpoke Fails if we cannot poke the element at the specified index, or if the index is out of bounds.
executes some basic tests
This program is licensed under the GNU Lesser General Public License v3 or later. Note that the LGPL is a set of additional terms that apply on top of the GPL. The text of the GPL3 is provided in LICENSE.GPL3, the text of the LGPL in LICENSE.LGPL3.