If you bought the paperback after 26th January 2018 or the Kindle version after 3rd March 2018, all these corrections and clarifications have been made in your copy. You may still find the discussion interesting, including the links, which you may want to follow up.
Now that I have retired from the contract consultancy world, I have had more time to read and to find out what is really going on without so much distraction from the marketing machinery. I was aware that Professor Michael Stonebraker was still active in the field, but I had no idea how busy he continues to be, nor did I appreciate the large pool of expertise that has been built up through his team at MIT CSAIL. There are now professors scattered through the top US universities, and in other countries, who have studied and carried out research with Michael Stonebraker, and this team has come up with some genuinely valuable innovations.
Unfortunately the market has picked up some of the words used to describe these innovations and has misused these words, as it always does, to increase sales without delivering real innovation. For every great product, there are several others that claim the same buzzwords but fail to deliver the substance.
I am sure that there will be more corrections and clarifications to my book, but this first set comes from a few weeks reading papers produced by Stonebraker's team over the last decade or so. It is interesting to look back this far, because we can find insightful analysis of emerging trends, with a remarkable number of accurate predictions about what has happened since. We also can witness the birth of some really smart products.
One of the themes that have driven Stonebraker's group in this period is One Size Does Not Fit All. You should read the paper (or at least watch the interview), but I'll summarise the message here.
Relational databases have been used for several different classes of applications:
Some of these work well with relational databases and some less well. OLTP databases have been very successful. In this area almost all commercial and public-sector systems run on relational databases. Data Warehousing has had some big successes and a number of projects that never really delivered what they promised, or that did so by spending a lot more money than anticipated. Stream processing and text processing have been weak areas for relational databases. Small databases can work well, but the level of knowledge required to build them is a bit more than using a spreadsheet (especially if you have already learned to use a spreadsheet) so they get built the easy way and then they often fail to deliver reliability, accuracy and scalability.
Stonebraker and his team have looked at these. One area that particularly interested me was the conflict between OLTP systems and data warehousing systems. OLTP systems need very high reliability and the ability to run thousands of short transactions every second. Data warehouses need fast bulk loading and high performance read-only queries on very large data sets.
For data warehouses Stonebraker concluded that a specialised database that took advantage of massively parallel processing (MPP) and used "column-based" rather than "row-based" storage would be the right approach.
I experienced both, separately.
Netezza was a superb MPP system. I first experienced it at a leading mobile phone company where it gave query times on average 170 times better than Oracle on seven billion mobile phone call records and all the associated data. Oracle was still used for the phone billing system because Netezza could not cope with that.
Oracle and Netezza each provided specialised functions, both within the relational database model and both accessed via SQL. The specialisation gave the best results.
Later I worked on a data warehouse at a very large Turkish Bank. This used a column-based database - Sybase IQ - and again gave excellent results. This bank also used Oracle for its OLTP systems.
The combination of MPP and column-based storage is something that Stonebraker's team pioneered to give what is now the ultimate data warehouse performance, still using the relational database model and still accessed using SQL.
For OLTP databases Stonebraker's team took a more radical approach. They considered the changing characteristics of modern hardware, including performance and cost, and concluded that an OLTP database, which is always going to be much smaller than a data warehouse in the same business, could usually run entirely in memory. This enabled them to cut out large and complex areas of the general purpose database, and produce a database engine that runs at very much higher transaction rates, still with the reliabilty and safety of a conventional relational database. The resulting product is VoltDB. (They don't mess about - when they have a good idea they build a product to prove it) The overall paper behind VoltDB is readable and very interesting.
Stonebraker's team continued to address the other uses of databases and has produced more very promising solutions. If there is any other group that has done anything nearly as exciting as this in the database world, I want to know about it. I have not found such a group yet.
In my book (primarily in Chapter 17 - Big Data) I used "Big Data" (capital B, capital D) to be synonymous with Apache Hadoop and the "ecosystem" of tools that go with it. This is the way the term is used in the commercial world, for the most part. It reflects the relentless marketing of companies such as Cloudera, Hortonworks, MapR and others who like to shout that Hadoop-based systems are the only solution for big data.
Many people who carry out research into big data systems, and many people who work on commercial big data applications, know that there are other established and emerging solutions to manage very big data. Smart companies that really do have big data are now moving away from Hadoop and MapReduce. The noise from the Hadoop/MapReduce companies is just noise; the products do not live up to the hype. Fortunes have been made by the vendors, but the companies that believed the hype and invested heavily in these products have suffered increased costs and sub-standard solutions.
Very sadly, in my view, some respectable database companies have felt obliged to jump on the Hadoop band-wagon. Fortunately, in my view, the wheels appear to be falling off that band-wagon.
As I hope I make clear in the book, there are other, better ways to handle big data.
In Chapter 14 - Tuning I have a go at so-called "in-memory" databases. I was not aware of VoltDB then. VoltDB does provide multiple innovations to use memory to make massive gains in transaction processing speed, and it is backed up by really solid research. It appears to be the exception, so for the others I stand by my complaint. I remain unconvinced by the "in-memory" claims of Exasol, TimesTen, eXtremeDB, SAP Hana, solidDB, etc.
In Chapter 18 - Database Features To Be Wary Of I advised against writing complete applications in the database stored-procedure language. I stand by that advice.
I gave an example of what I consider to be good use of stored procedures (or user-defined functions which are one type of stored procedure). I failed to mention another good use of stored procedures, and that is as a tuning option for high-volume transaction processing systems. This does fit in with my suggestion that we should collect all the data necessary for a transaction and then submit it to the database without any intervening processing, so that each transaction is completed as fast as possible. I could also say that it is covered by "Advanced Database Tuning" on page 291 in Chapter 14 - Tuning, but the truth is that I overlooked it, despite a nudge from my good friend Mike Tanner.
In Chapter 1 - Learning to Drive Your Database on page 45, I suggested that Ted Codd and Chris Date were joint inventors of the relational database. In fact, Ted Codd was the inventor and Chris Date documented the concept in An Introduction to Database Systems and many other publications. Codd and Date worked together at IBM in the early days of relational databases, and again later in the Codd and Date Consulting Group.
Page 136 in Chapter 6 - Data Types is about the Boolean data-type. In practice this applies only to PostgreSQL and some other databases derived from
In Chapter 6 - Data Types, on page 139, in the Complex Data-Types section I talked about timestamps. I failed to mention that this data-type is called