It sounds like what he wants is some sort of userland swapper. In this
case, the implementation would be to decompress when pages are swapped in,
and simply drop the page when it's swapped out.
Given the current constraints, and the fact that decompression will touch
the entire dataset _anyhow_, it would make sense for the decompression pass
to prime a data structure with pointers to non-zero data within each page
(probably int-aligned for performance reasons), and mark it disposable as
suggested elsewhere. Skip any page which is all zeros. Then when the data
is to be used, mlock() it, check to see if any of the non-zero pointers now
point to zeros, decompress those pages as needed, blit them, munlink(), and
mark them disposable again.
Actually, that might be better than a userland swapper, because in that
case there's nothing to prevent you from blitting half the dataset, and
then hitting a swap.
----- Original Message -----
From: Matthew Dillon <email@example.com>
To: Kevin Day <firstname.lastname@example.org>
Cc: Kevin Day <email@example.com>; Daniel C. Sobral <firstname.lastname@example.org>;
Sent: Thursday, September 23, 1999 9:54 AM
Subject: Re: Idea: disposable memory
> :I'm now playing with compressed data streams. The decompression is slow,
> :I'd like to cache the *decompressed* version of these files. I end up
> :allocating large amounts of ram in one process to cache the decompressed
> :data. This is a disavantage over the above scenario, since now the
> :swaps out my decompressed data when more ram is needed elsewhere.
> :out then swapping back in my decompressed data is about 4x slower than
> :re-reading my compressed stream and decompressing it again.
> :Why don't I just allocate a predefined amount of memory and use that for
> :cache all the time? Most of the time we have about 20MB free on our
> :Sometimes we end up with about 2MB free though, and what's happening now
> :that I start paging out data that I could recreate in less time than the
> :page-in/page-out takes.
> Hmm. Well, you can check whether the memory has been swapped out
> mincore(), and then MADV_FREE it to get rid of it (MADV_FREE'ing
> that has been swapped out frees the swap and turns it back into
> That doesn't get rid of the swapout bandwidth, though.
> I think, ultimately, you need to manage the memory used for your
> manually. That means using mlock() and munlock() to lock your cache
> memory. For example, choose a cache size that you believe the system
> can support without going bonkers, like 5MB. mmap() 5MB of ram and
> mlock() it into memory. From that point on until you munlock() it or
> exit, the memory will not be swapped out.
> If the purpose of the box is to maintain the flow of video, then the
> is a critical resource and should be treated as such.
> Matthew Dillon
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message