This seems like an interesting move from Oracle. Times Ten’s Cache is an in-memory cache type database that works very much like MySQL Cluster – with read/write functionality – along with the ability to “divert” to a backend (Oracle) database when the data needed is not memory resident.
It will be interesting over the next few months to see what Oracle does with this. On the surface it’s quite obvious that it’s an attempt to get their foot in the door with the likes of telcos and financial services companies, for their “frontline” systems, where they are rather lacking due to the cumbersome nature of the beast for “real time” data.
So you may have read this through the MySQL category on my blog, and wondered “How the hell does this relate to MySQL”? As any seasoned MySQL user would know, MySQL Cluster has been around for a while now, and whilst it’s great for “smaller” apps – that require very fast “real time” access, but don’t necessarily store huge amounts of data – many have noted the serious lack of any disk based “second stage” storage for older data that may not be used very often. MySQL Cluster is already geared towards the areas where Oracle is trying to get to right now through acquisitions. MySQL Cluster however has fallen down on the VLDB back end requirements of some of these systems.
Until 5.1 it seems. Today I saw a commit to the internals email list from Pekka Nousiainen that refers to “Pgman” – which seems to handle buffer pool caching and flushing from/to disk based storage – for the NDB storage engine. I’ll save you all the code, but this comment seems to sum it up fairly well for now:
+ * PGMAN
+ * PAGE ENTRIES AND REQUESTS
+ * Central structure is “page entry”. It corresponds to a disk page
+ * identified by file and page number (file_no, page_no).
+ * A page entry is created by first request for the disk page.
+ * Subsequent requests are queued under the same page entry.
+ * A new entry must wait to be “bound” to an available buffer page
+ * (called “real page” here). If the disk page is not “new” (empty),
+ * the entry must also wait to be “mapped” via “pagein” from disk.
+ * Entries are released on demand when page requests arrive for unknown
+ * pages. Release candidates are entries which point to a disk page
+ * which is “clean and not used”. They are ordered by some variant of
+ * least recently used (LRU).
+ * A background clean-up process makes “dirty” pages clean via “pageout”
+ * to disk. UNDO log entries are first flushed up to the maximum log
+ * sequence number (LSN) of the page. This is called write ahead
+ * logging (WAL). The clean-up process prefers “dirty and not used”
+ * pages and lower LSN values.
+ * A local check point (LCP) performs complete pageout of dirty pages
+ * since given LSN. It needs a stable entry list to iterate over.
+ * Page entries are put on ordered lists accordingly:
+ * The “main” list contains all entries. It is ordered by its unique
+ * key (file_no, page_no). It is used to look up page entries. LCP
+ * iterates over it to cover all candidate pages.
+ * Each entry also belongs to exactly one “sublist”. These drive page
+ * processing. The sublist is determined by page “state”. The state
+ * contains page properties such as discussed above.