A registry has several technical components. Before trying to integrate them, let's enumerate what they are.
"database" does not imply that it runs on a DBMS. I just mean a way (may be even a simple text file with some procedures to edit it properly) to store the domains and their name servers.
Unless you run a "thin" registry, you will need to store the social information, which is the names and addresses of domain names holders and domain contacts (technical, administrative, etc).
For both databases, writing the schema is not obvious and you can use the one in OpenReg as a starting point. Do note that the choice of a schema raises a lot of policy issues.
It is highly recommended that this database is physically the same as the domain database before, in order to avoid inconsistencies. But some registries keep two separate databases, often for historical reasons. In that case, people often call it the "whois database" because of the protocol that historically served it.
You need to export the database in a format which is suitable for your name servers. Of course, you can always use the name server's zone file as a "low tech" database, which is possible for a small registry. Or you can plug your name server directly into the database which is possible with PowerDNS. But in all the other cases, you will need a (small) program to export the database.
You will need a way for the people working at the registry to read the database(s), and to update it. At the present time, it means a Web interface, usable from any Web browser. Die-hards users could still use the SQL command-line interface :-)
Since it may be a complicated program, the use of a Web Application Server, such as Zope may be a good idea.
People outside of the registry will need to read and update the database, too. I see three cases, in order of ascending size:
A very small registry can use low-tech interfaces (phone and paper) to receive domain updates and then use its internal interface.
A small or medium-sized registry will have no "registrars", no intermediaries. It will provide directly the registrants with a set of Web pages to create and manage their domain names.
The ease of use of this Web application is paramount to limit the load on the support team.
A large registry will probably have registrars and will offer them an interface with a protocol like EPP or RRP (RFC 2832) or a simple key-value template like the RIPE-NCC protocol or the CORE payload protocol.
Unless your domain is entirely hosted outside of the registry, either by friends or by a commercial company, you will need name servers and you will need to monitor and maintain them.
If the creation of a domain involves several steps (such as some manual checks), you may need a ticketing system, able to give an unique ID to each transaction, allowing easier referrals. This tool will give you the possibility to track work in progress.
Such a tool is also useful to manage things like operational tasks to perform (software installation, for instance), and customer support requests.
The recommended tool is Request Tracker but many other similar tools exist in free software.
It is highly recommended to have a whois server (RFC 954) to distribute social information (such as the technical contact) about your domains. In the future, other protocols may replace whois.
Of course, you will need backups. The complete destruction of the database would be a catastrophe. Therefore, it is highly recommended, not only to make backups on permanent storage (one CD-ROM is more than enough for most domain registries), but also to store them in far away places, to guard against catastrophic failures such as fire or war.
To do so, a disk-to-disk backup is often the simplest solution when the two machines are connected by the Internet. You can replicate your database automatically (this is very easy to set up with MySQL), or simply copy the data with a program like rsync (this can be automated, see for instance this documentation. It is easy to automate these operations.
If you want to register domain names in other character sets than US-ASCII, for instance with the Arabic alphabet, you will need some IDN tools.
It is recommended that all the components of the registry reads from and updates to the same repository. This will avoid inconsistencies. To achieve this goal, you can either:
Use one database, which will be the authoritative source of information for everything.
Use a client-server middleware, for instance with protocols like XML-RPC, that all your applications will use.
You can have a common library, used by all your programs, to read and update the datanase(s).
The only available free software at the present time, to run a registry, is OpenReg. It provides an EPP interface, a middleware program (so you can add your own programs), a whois server, a database (PostgreSQL by default).
If you use a DBMS to store the data, you can use either PostgreSQL (which runs ".org" and ".info") or MySQL.
You will need some programming skills, anyway, because all registries are different and it is likely you will have some specific requirments. "Scripting" languages like Perl or Python are necessary for a registry.
Even if you limit your requirments, many software that are available will need some sort of glu to hold together.
Here is a partial list of the components that are not, as far as I know, available as free software:
The internal Web interface of a registry.
The external Web interface of a registry, for the registries that have no registrars and therefore have to provide such an interface.