Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

[DEV] dbms3.r 01

 [1/50] from: g::santilli::tiscalinet::it at: 13-Jan-2002 15:36


PROJECT NAME: dbms3.r (temporary) PURPOSE: writing a small and simple relational database management system in REBOL, for use with applications that don't need a full blown separate DBMS such as MySQL. With respect to dbms.r, it will be more efficient in the data storage to be usable with much more rows per table, and possibly will have a more powerful/abstracted interface. DEVELOPMENT, part 1 let's start with the low level functions to handle tables on disk. table on memory could be useful too, but should be easier to implement. we can then adapt dbms.r to use the new format, to have an usable system in less time. we'll need a better way to store rows, maybe in blocks or all in one file. the problem with using one single file for the data is that currently rebol does not allow for easy seeking on direct ports. subdividing the table in blocks of 100 or 1000 rows and having a file per block might be more efficient - maybe someone wants to run some benchmarks. if we use one file, we need a file to index the rows in the file since i expect to have dynamic sized rows as in dbms.r. the index can be kept in memory too, since it will be a block of integers, so about 16 bytes per row. we're not targetting 1'000'000 rows per table so that does not seem a problem. i think we can proceed this way: let's create all three kinds of tables, then we can let the user choose. let's start with tables on memory, that can be simply represented with blocks. our interface should include ways to: append a new row, pick a row, change a row, remove a row. -- ; brett, i like your idea. :) memory-table-funcs: context memory-table-funcs-code -- ; i like joel's style, but i'll stick with the standard here memory-table-funcs: context [ ; memory-table-funcs-code row-append: func row-append-spec row-append-code row-pick: func row-pick-spec row-pick-code row-change: func row-change-spec row-change-code row-remove: func row-remove-spec row-remove-code ] -- memory-table-funcs: context [ ; memory-table-funcs-code row-append: func [ "Append a row to an on-memory table" table [object!] row [object!] ; returns the row-id of the new row ] row-append-code row-pick: func [ "Pick a row from an on-memory table" table [object!] row-id [integer!] ; returns the row or none! for an invalid row-id ] row-pick-code row-change: func [ "Change a row in an on-memory table" table [object!] row-id [integer!] changes [block! object!] ; returns the row-id of the changed row, or none! for ; an invalid row-id ] row-change-code row-remove: func [ "Remove a row from an on-memory table" table [object!] row-id [integer!] ; returns true, or false if the row-id is invalid ] row-remove-code ] -- some comments: the row-id will be the index of the row in the block, so it's an integer. this might be different for other kind of tables. also, the row-id of a row is not guaranteed to remain constant for a row, like in dbms.r. the changes argument of the row-change function can be a block or an object; a block is useful for things such as [column: column + 1] where you want to change the value of a column based on its previous value. [TO BE CONTINUED] (I'll stop here for today, as I want too see the reaction of the list before going too much further.)

 [2/50] from: petr:krenzelok:trz:cz at: 13-Jan-2002 16:41


Hello, just two notes - I think that we also need some library wrappers. If you will look at some open sourced libraries, you'll find plenty of various languages mentioned, except Rebol of course. - next Rebol update will feature call-back functions in Library interface - next Rebol update should also introduce new port type, with ability to perform seek upon direct opened port, maybe you would want to wait for this one? The problem with native Rebol dbms is - robustness and stability - locking, several processes working with one db at once, rollbacks, etc. stuff. Have you looked for e.g. at mySQL embedded? It should be just some kind of linkable library, so really small, available currently for many platforms. Maybe we could adopt that one? It features the same interface as full mySQL version ... *libmysqld* makes MySQL suitable for a vastly expanded realm of applications. Using the embedded MySQL server library, one can embed MySQL into various applications and electronics devices, where the end user has no knowledge of there actually being an underlying database. Embedded MySQL is ideal for use behind the scenes in internet appliances, public kiosks, turn-key hardware/ software combination units, high performance internet servers, self-contained databases distributed on CD-ROM etc. Many embedded MySQL users will benefit from the *dual licensing scheme* of MySQL, where besides the GPL license also commercial licensing is available for those not wishing to be bound by the GPL. The embedded MySQL library uses the same interface as the normal client library, so it is convenient and easy to use. http://www.mysql.com/doc/l/i/libmysqld.html PS: of course I don't want you to leave your current thoughts and work, just a suggestion to look at external solution not necessarily reinventing the wheel ... Cheers, -pekr-

 [3/50] from: joel:neely:fedex at: 13-Jan-2002 11:54


Hi, Gabriele, Let me first offer a word of support to your approach. As is probably obvious to anyone who has read the first refactoring article on REBOLforces (hint, hint, hint ;-) I strongly believe that there's much more value in seeing HOW code is designed and constructed -- including false starts, bad ideas, backtracking, refactoring, etc. -- than in simply having a finished product. At the same time, it takes real courage to expose one's thinking in near-real-time to an audience or a group of collaborators. I applaud your bravery! Gabriele Santilli wrote:
...
> DEVELOPMENT, part 1 > > let's start with the low level functions to handle tables on disk. > table on memory could be useful too, but should be easier to > implement... > > i think we can proceed this way: let's create all three kinds of > tables, then we can let the user choose... >
Good plan IMHO. Whever you come to a fork in the road, take it! Seriously, I've seen lots of great designs come from a decision to allow downstream decision making rather than wiring one set of assumptions into the fundamental design.
> ... let's start with tables > on memory, that can be simply represented with blocks. our > interface should include ways to: append a new row, pick a row, > change a row, remove a row. >
...
> some comments: the row-id will be the index of the row in the > block, so it's an integer. this might be different for other kind > of tables. also, the row-id of a row is not guaranteed to remain > constant for a row... >
My experience suggests that there's value in three kinds of access to elements of a collection, which I'll call by semi-arbitrary names for this discussion: 1) "physical" access -- corresponding to your notion of "row-ID" in which traversal of a collection is done in the fastest possible manner -- SINGLE VALUED. 2) "identity" access -- via a unique ID which is guaranteed not to change throughout the lifetime of the collection (a sort of "true name" of each entity) allowing references to entities in the collection to remain valid even if the order or structure changes -- SINGLE VALUED. 3) "criteria-based" access -- via a search expression, which may refer to one or more attributes of an entity, possibly represented as a function passed to an iterator which finds and presents all entities possessing the specified property(ies) -- SINGLE OR MULTIPLE VALUED. I'd suggest that "row-ID" be as thoroughly hidden as possible, e.g., only used within a loop that is scanning the collection WITHOUT the possibility of modifying the collection or any of its elements in a way that would cause the meanings either of "current position" or "positions already visited" to change. Otherwise (e.g., one entity that refers in a persistent way to another entity) a "unique ID" should be used. Just my $0.015 of the moment ;-) -jn- -- ; sub REBOL {}; sub head ($) {@_[0]} REBOL [] # despam: func [e] [replace replace/all e ":" "." "#" "@"] ; sub despam {my ($e) = @_; $e =~ tr/:#/.@/; return "\n$e"} print head reverse despam "moc:xedef#yleen:leoj" ;

 [4/50] from: rotenca:telvia:it at: 13-Jan-2002 19:20


Hi Gabriele,
> PROJECT NAME: dbms3.r (temporary) > PURPOSE:
<<quoted lines omitted: 4>>
> rows per table, and possibly will have a more powerful/abstracted > interface.
I like it. I want it. I feel the need of a standard interface. It must be short, closed in itself and fast (and bug free of course :-). In other cases it is a better choise, i think, to write own routines and integrate them in the program. --- Ciao Romano

 [5/50] from: g:santilli:tiscalinet:it at: 13-Jan-2002 19:57


Hello Joel! On 13-Gen-02, you wrote: JN> At the same time, it takes real courage to expose one's JN> thinking in near-real-time to an audience or a group of JN> collaborators. I applaud your bravery! Well, I hope you will all be indulgent to me. :) JN> My experience suggests that there's value in three kinds of JN> access to elements of a collection, which I'll call by JN> semi-arbitrary names for this discussion: JN> 1) "physical" access -- corresponding to your notion of JN> "row-ID" JN> in which traversal of a collection is done in the fastest JN> possible manner -- SINGLE VALUED. This is the case for the low-level functions. JN> 2) "identity" access -- via a unique ID which is guaranteed JN> not to change throughout the lifetime of the collection JN> (a JN> sort of "true name" of each entity) allowing references JN> to JN> entities in the collection to remain valid even if the JN> order JN> or structure changes -- SINGLE VALUED. This is why I used them in dbms.r; but now I'd like to abstract more and let the user decide the keys for a table. Anyway, as things proceed, I might decide to include some kind of special key to optimize speed. JN> 3) "criteria-based" access -- via a search expression, which JN> may refer to one or more attributes of an entity, JN> possibly JN> represented as a function passed to an iterator which JN> finds JN> and presents all entities possessing the specified JN> property(ies) JN> -- SINGLE OR MULTIPLE VALUED. This is what the user sees, even in the current dbms.r. JN> I'd suggest that "row-ID" be as thoroughly hidden as JN> possible, e.g., only used within a loop that is scanning the The plan is that it will be used in indices only, internally. If a row gets moved for some reason, the index can be updated; of course I think such change will not be very frequent, but mainly originating from an optimization of the table or something like that... JN> Just my $0.015 of the moment ;-) Thanks! They are much appreciated. :) Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [6/50] from: g:santilli:tiscalinet:it at: 13-Jan-2002 21:06


Hello Romano! On 13-Gen-02, you wrote: RT> I feel the need of a standard interface. It must be short, RT> closed in itself and fast (and bug free of course :-). In RT> other cases it is a better choise, i think, to write own RT> routines and integrate them in the program. Have you seen dbms.r? I plan to keep a similar interface (well, better, I hope :). Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [7/50] from: rotenca:telvia:it at: 13-Jan-2002 23:13


Hi Gabriele,
> Have you seen dbms.r? I plan to keep a similar interface (well, > better, I hope :).
No, i'll see asap. --- Ciao Romano

 [8/50] from: rgaither:triad:rr at: 13-Jan-2002 17:35


Hi Gabriele,
>PROJECT NAME: dbms3.r (temporary) >PURPOSE:
<<quoted lines omitted: 4>>
>rows per table, and possibly will have a more powerful/abstracted >interface.
First, I would like to add my support of such an open development effort! :-) I am going to stop right here for a moment though and ask some tough questions about this project. 1. Do we really "need" another RDBMS? 2. Is creating a RDBMS playing to the strengths of REBOL? 3. What is the problem you are trying to solve? I am playing devils advocate here because I want much more than a RDBMS. I'd like something more along the lines of an OODBMS or perhaps even something completely different. I do also want the foundation of something that can become multi-user, distributed, and work with the strengths of REBOL. It seems to me :-) the very first thing we should be building is a layer of functions, objects, or a dialect that deals with binary data. Not already into the functions of a RDBMS or an OODBMS but just the basic ability to manage blocks of binary information. This layer of managing blocks is key to building any type of binary database, for whatever kind of purpose or structure. I have some design notes on just such a layer that I will cleanup and throw into the mix later today. Thanks, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [9/50] from: rotenca:telvia:it at: 14-Jan-2002 0:31


Hi Gabriele, Now i've started to read the dbms code. The first notes: 1) I like to have a function to update the db. Until the funtions call all the changes should resides in memory or in a temp file. 2) I do not like to use a dir for a db. Neither to use a dir name for a db name. My preference is for basename. 3) I should like a db-select dbname none 4) I do not like 1 record 1 file: on Windows cluster are of 32-64kb (what a beatiful thing!) but i know changing this is your actual first target. 5) Do you keep in memory Columns and indexes? --- Ciao Romano

 [10/50] from: rotenca:telvia:it at: 14-Jan-2002 0:41


I can't answer for Gabriele, but i can say why i need it: after writing 3 different custom db for my programs, i feel the need of a "standard" rebol db to store/retrieve data. I do not want to use an external db only for some items. I was thinking to write my own but if Gabriele write it i'll use it (and it will be better than mine, i'm sure) It must be small (and fast) because it should be included in other code. It should not waste gc words space. I want only basic feature. I think that the select dialect of dbms is already too much complex. --- Ciao Romano

 [11/50] from: rgaither:triad:rr at: 13-Jan-2002 19:09


Hi Romano,
>I can't answer for Gabriele, but i can say why i need it: after writing 3 >different custom db for my programs, i feel the need of a "standard" rebol db >to store/retrieve data. I do not want to use an external db only for some >items.
Don't get me wrong - I need it as well. I just don't want to start with rows and columns and having to think about my REBOL data within the confines of a RDBMS. I am a database developer by day and a reboler by night but I know REBOL is capable of a very different kind of database.
>I was thinking to write my own but if Gabriele write it i'll use it (and it >will be better than mine, i'm sure)
I too have thought about it and can easily say either you or Gabriele could write one in REBOL better than I. Most on this list could I believe. :-)
>It must be small (and fast) because it should be included in other code. It >should not waste gc words space.
Small and flexible are more important to me than fast.
>I want only basic feature. I think that the select dialect of dbms is already >too much complex.
I don't mind some complexity as long as the design is built to manage most of it internally. Just my opinons mind you. If a RDBMS is what Gabriele needs and wants to build I will be glad to help in whatever ways I can. I just want to open up the discussion a bit in other directions to see if there is any interest. Thanks, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [12/50] from: rotenca:telvia:it at: 14-Jan-2002 2:12


Hi Rod,
> I just don't want to start with rows and columns and having to think > about my REBOL data within the confines of a RDBMS. I am a database > developer by day and a reboler by night but I know REBOL is capable > of a very different kind of database.
The advantage of row/col is for export functions. It can be easy converted in a CVS db for more complex works or to change direction if you see that db becomes too big. But i'm open to every solutions. In my custom db i use nested blocks which are not easy converted in a row/column struct.
> Small and flexible are more important to me than fast.
I think it could be modular: basic functions in the main script, additional functions, also the same but more complex, in a second one compatible. About velocity, i had problems until i started using hash! for indexes.
> I don't mind some complexity as long as the design is built to manage > most of it internally.
I think about dimensions. But it is a fixation of mine. --- Ciao Romano

 [13/50] from: rgaither:triad:rr at: 13-Jan-2002 20:52


Hi Romano,
>The advantage of row/col is for export functions. It can be easy converted in >a CVS db for more complex works or to change direction if you see that db >becomes too big. But i'm open to every solutions. In my custom db i use nested >blocks which are not easy converted in a row/column struct.
There are many advantages to a row/col based design. I don't mean to preclude it but I want to also do nested blocks or complex objects or even simple name:value pairs.
>> Small and flexible are more important to me than fast. > >I think it could be modular: basic functions in the main script, additional >functions, also the same but more complex, in a second one compatible. About >velocity, i had problems until i started using hash! for indexes.
Modular is good and having different functions that match the task in complexity also sounds good. This is why I want to start at the bottom - the interaction with a binary file. Then, on top of that we could build functions to manage rows and columns, or objects, or name:value pairs and so on.
>> I don't mind some complexity as long as the design is built to manage >> most of it internally. > >I think about dimensions. But it is a fixation of mine.
You would be right at home with the Pick/UniVerse database system then. It is multi-dimensional in nature. I am however just an relational db guy who wants more options than the traditional ones provide. I also want it all to be simpler as well. (Can you say dreamer? :-)) Thanks, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [14/50] from: jason:cunliffe:verizon at: 13-Jan-2002 13:40


Gabriele your project is very interesting both for its intended output and approach. Vanilla Proposal My first comment is about achieving the level of collaboration hoped for. I have been experimenting a lot recently with Vanilla. It is a wonderful system, offering an excellent platform for all sorts so of joint creative projects because it fuses the best features of wikis with weblogs. Because Vanilla is written in REBOL, it is fast and fun work with, and of course a natural fit for REBOL dev projects. Pesonal contribution My REBOL programming skills are minimal, and I know little about dmbs design, so I am not likely I can contribute much directly there. But I am very willing to setup and maintain a Vanilla site for dbms3r project. Hopefully in time the site itself could become a useful testbed for dbms3r. I am committing this year to a Vanilla-based creative interfaces, so my effort is already focused there. My initial targets for Vanilla development includes: 1 - css and imaging extensions supporting a visual matrix toolset for artists and filmmakers 2 - REBOL source code display and edit [embed colorize, makedoc pro etc] 3 - client-side tools in REBOL/View and Flash5 for easy upload, management, special display 4 - documentation and special admin [transfer between site, extended snip management ] 5 - alternative input/output such as email-to-Vanilla Longer term, more ambitious effort will focus on: 6 - peer-to-peer, site-to-site Vanilla interactions [using rugby / xmlrpc ?] 7 - scalability and strong database options [perhaps dbms3r ??] 8 - remote control and presentation dynamics 9 - dynamic whiteboard 10 - map + geographic extensions Server My server is down this week for rebuilding [RedHat 7.2]. Hopefully it will go smoothly and I can put up a stub site very soon to see how people feel about the idea in practice. A natural by-product might be to complement the resonant pearls-of-wisdom approach of Code Conscious. Risks - wikitosis There are problems with wikis - imo they need to be tamed in precise ways, their features and design presentation need to fulfill to people and projects much more. I like the idea of them in theory, but hate most of them in practice. They are fascinating techno-political instruments offering a glimpse of reclaiming the orginal collaborative promise/visions of 2-way www [1990] One reason I like Vanilla so much is that it offers rapid ways to test, tune and compare the systems behavior. - list culture vs. wiki cultcha Also how to make such sites cohesive to mailing lists like this and actual source practice.? Advantages All REBOL showcase fast fun development community resource
> The problem with native Rebol dbms is - robustness and stability - > locking, several processes working with one db at once, rollbacks, etc. > stuff. Have you looked for e.g. at mySQL embedded? It should be just > some kind of linkable library, so really small, available currently for > many platforms. Maybe we could adopt that one? It features the same > interface as full mySQL version ...
hmm... As I understand it, the main point of Gabriele's dbms3r is to offer REBOL database solution without any external dependencies. Integrating libraries has many benefits, but risks complexity of installation and presently has callback issues. Holger wrote that the library callback issue will be answered in next release. That leaves questions of reinventing bicycle wheels in Siberia...or not BerkeleyDB is another option to MySQL. Its available on many platforms, a rich set of features and API wrappers in many languages. I have used it a little with Python and Zope and liked it very much. http://www.sleepycat.com/ Python and Zope have been developing generic syntax for database use to allow Python developers freedom of choice and scalability with regard to the actual underlying database library. This seems like a great strategy. A problem may be very simple to begin with. So for sake of rapid prototyping and easy installation a developer uses the local generic db toolset. Later if needed can make the switch. I don't know how well this really works in practice. It certainly aids consistency in learning and programming and hopefully makes it more 'client' friendly. What can dbms3r learn from Python's db interfacing ? ./Jason

 [15/50] from: rgaither:triad:rr at: 13-Jan-2002 22:13


Hi Gabriele,
>PROJECT NAME: dbms3.r (temporary) >PURPOSE:
<<quoted lines omitted: 4>>
>rows per table, and possibly will have a more powerful/abstracted >interface.
Here are some of my design thoughts at the binary DB level before getting into the row/col level functions. I am assuming (dangerous I know) that binary storage is an option at all. If you want a text based storage system then ignore this input. :-) I am concerned in my ignorance about your comment on REBOL not being able to seek "easily" on direct ports. Could you expand on this for me? It may be that REBOL just can't do the block oriented binary DB that I am envisioning - at this version anyway. Please note - these design notes are just that. They only represent some effort at considering the requirements of a binary DB storage system. It is based on some C work I did for a handheld computer I was programming for a couple of years ago. In the end we kept our application data in csv style text files for simplicity so none of this made it past "design". 1. Every database has a leading "Master" block. This block contains structured information about the database so it can be used properly. Master_Block Type Version Special Key Set of Flags Block Size Extend Count Free Block Count Free Block Root Chain Block Root 2. The rest of the database are "Normal" blocks of different types. Each of these blocks contains a standard fixed length "Header" with structured information about the block. After the header is the data storage area. Block_Header Key (DB location by offset based on fixed block size) Type Datatype Set of Flags Next Block Key Previous Block Key
>From these basic pieces everything else is built. Each block in the database
can belong in a doubly linked list style structure to provide a "series" of blocks. Special types of "Named" series such as an Index or Hash Table can be built by registering them with the list of chains stored in the root chain block. The block manager is only responsible for the following activities: 1. Creating, opening, and closing the binary file. 2. Initilizing and managing the master block data. 3. Providing functions to manage named chains. 4. Reading and writing blocks. 5. Managing the free block chain for allocation/reuse. 6. Traversing the block chains. At only a slightly higher level you would make use of the block chain structures to fill purposes such as defining a table to storing data and indexing it. Above that is where you would start interacting with the database as rows/cols or whatever it was designed for. I am leaving out lots of details for brevity and at this point want to follow Gabriele's lead and stop here and wait for reactions. I don't want to run off in a direction no one is interested in or distract from the main thread any more than I have so far without some list input. Thanks for listening, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [16/50] from: rotenca:telvia:it at: 14-Jan-2002 16:56


Hi, Gabriele, another wish of my long long long list :-) I do not like to make a pick for every row id returned by db-select, neither a db-foreach. I should like to ask something like: select [name surname] [sort by [surname name]] ;rebol sql? to have a block: [["joe" "abbitt"]["allen" "callaghan"]] I can do it with db-foreach, but i think is slow to convert data in object and then in block. I feel that block are the best way to go with db in rebol or at least i use always blocks, because of View which requires block for list, textlist and so on. Perhaps could exist two series of interfaces: blocks and objects or perhaps it is already possible in dbms and i did not see it. (i say everything i can wish/think, hope not to exaggerate) --- Ciao Romano

 [17/50] from: rotenca:telvia:it at: 14-Jan-2002 17:06


Hi Rod,
> There are many advantages to a row/col based design. I don't mean to > preclude it but I want to also do nested blocks or complex objects or even > simple name:value pairs.
Now i realize that my Rebol databases are made of nested blocks of variable length. It is not clear to me how to convert them in a row/column scheme. Any ideas?
> Modular is good and having different functions that match the task in > complexity also sounds good. This is why I want to start at the bottom - > the interaction with a binary file. Then, on top of that we could build > functions to manage rows and columns, or objects, or name:value pairs > and so on.
Now i understand your point of view. Are you sure it can be done without any data structure representation?
> You would be right at home with the Pick/UniVerse database system > then. It is multi-dimensional in nature. I am however just an relational > db guy who wants more options than the traditional ones provide. I also > want it all to be simpler as well. (Can you say dreamer? :-))
Dreams are free. :-) --- Ciao Romano

 [18/50] from: rgaither:triad:rr at: 14-Jan-2002 12:38


Hi Romano,
>> There are many advantages to a row/col based design. I don't mean to >> preclude it but I want to also do nested blocks or complex objects or even >> simple name:value pairs. > >Now i realize that my Rebol databases are made of nested blocks of variable >length. It is not clear to me how to convert them in a row/column scheme. Any >ideas?
Yes, depending on what type of information you have "nested" the most likely option is to treat each nested block or set of blocks as a new table joined with the parent. The hardest part is if data from what would be the same related table appears multiple times, in different nesting locations. For example Addresses nested below individual Person data. Person 1 Home Address Work Address Person 2 Home Address Work Address It can be tough to know that the two different sets of address records are all part of the address table, just joined to different person records.
>> Modular is good and having different functions that match the task in >> complexity also sounds good. This is why I want to start at the bottom -
<<quoted lines omitted: 3>>
>Now i understand your point of view. Are you sure it can be done without any >data structure representation?
It certainly can't go very far before some amount of structure information must be imposed. Still, that is the second step and will vary with what is being represented. It is an important distinction even assuming you are building a RDBMS. For example storing index data as a B-Tree structure, or a Hash table, versus storing the row and column data. Each of the above still need to be converted into binary form and managed in variable sizes while reading and writing to the database.
>> You would be right at home with the Pick/UniVerse database system >> then. It is multi-dimensional in nature. I am however just an relational >> db guy who wants more options than the traditional ones provide. I also >> want it all to be simpler as well. (Can you say dreamer? :-)) > >Dreams are free. :-)
That is a really good thing for my finances! :-) Thanks, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [19/50] from: greggirwin:mindspring at: 14-Jan-2002 13:13


Rod, Romano, Gabriele, et al (This got longer than I intended, just so you know :) I can see the value in pure RDB systems, though I've rarely used them. Here's why. I'm a big believer in ADTs (Abstract Data Types) and the impedance mismatch between Types/Objects/Structs and RDBs is something I try to avoid at all costs. Yes, I have relationships in my data but most of them are either 1-to-1 or 1-to-a-few type relationships. The 1-to-many (which I call 1-to-infinite) case has been much less prevalent for me. In most of those cases, we had an ADT/object which acted as a kind of "relationship manager". A pure RDB model (I know I'm talking in absolutes here) doesn't allow for arrays (multi-valued values, which I call a 1-to-a-few relationship), which is far more common than the 1-to-infinite relationship, at least in my experience. I want to do as little work as possible, at the application level, to store and retrieve my data. I love the idea of a transparent persistent object system, with the ability to define relationships not implied by an object hierarchy, though most OODBMS I've looked at are far from transparent because they try to be all things to all people. A few years ago, I saw what I thought was a very well designed ORDBMS (orbject-relational) product. It was called Cascade/DB, from Microquill. Another company bought the product and it vanished from sight (and Microquill couldn't name the company who bought it, as part of the deal). It allowed for nested structures, nested tables, object references, and records were, IIRC, variable length blocks of data separated by newlines, so you could pull up the data in an editor without it being complete gibberish. I think editing it manually would have been disastrous, but you could at least see it. I don't know that the row/column format, as a native "mode", buys you anything. If you want to import/export data with other systems, I've always found it easiest to have specific tools for those specific systems. As soon as you get into nested structures, the model becomes burdensome when you try to generalize it. Again, in my experience, real-time interaction with other systems was rarely required. We did most data transfers in batch, polled, or manual import/export fashion. So, for me, I want a simple system that is tightly focused on storing and retrieving REBOL data (probably simple blocks, blocks of name/value pairs, and objects). It should handle up to maybe 50,000 records, though most cases will be far less I imagine. The little things I've done so far have actually worked well using native storage, but the scalability, robustness, and multi-user aspect are something I'm concerned about for more critical uses. --Gregg

 [20/50] from: g:santilli:tiscalinet:it at: 14-Jan-2002 21:49

Re: dbms3.r 01


Hello Rod! On 14-Gen-02, you wrote: RG> 1. Do we really "need" another RDBMS? Well... :-) (Ok, I choose the wrong project for the testbed of my idea. That was just because I'll need it soon... :) RG> 2. Is creating a RDBMS playing to the strengths of REBOL? Not necessarily. Look at my answers to Petr. RG> 3. What is the problem you are trying to solve? Imagine you are writing a software for a shoes shop. They'll need an archive of the shoes they have, and to record what they sell and what they buy for statistics etc. If it is a small shop, MySQL or similars might be overkill; you'll need something that is powerful enough (so that you can reuse code if you decide to switch to a MySQL client/server solution when thw shop grows or if you are contacted by another bigger shop etc.) but also small and simple so you have control on it (and the user does not have to install and maintain a MySQL server, you are not called because the user forgot to start the server and so on). That's my need. RG> I am playing devils advocate here because I want much more RG> than a RDBMS. I'd like something more along the lines of an RG> OODBMS or perhaps even something completely different. I do As I said to Petr, I have some ideas. (Actually, even the Information World can serve as an incredibly powerful distributed data base.) Nothing that can be useful right now anyway, and I'm quite busy, so... but we might discuss it if you are interested. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [21/50] from: g:santilli:tiscalinet:it at: 14-Jan-2002 22:19


Hello Rod! On 14-Gen-02, you wrote: RG> I am assuming (dangerous I know) that binary storage is an RG> option at all. If you want a text based storage system then RG> ignore this input. :-) Text based for simplicity, but... (see below) RG> I am concerned in my ignorance about your comment on REBOL RG> not being able to seek "easily" on direct ports. Could you RG> expand on this for me? It may be that REBOL just can't do the If you open a file port as /direct, you can only seek forward currently. This means you have to close the port and reopen it if you need to go back... This is expected to change in the next release. RG> block oriented binary DB that I am envisioning - at this RG> version anyway. Will just be slow currently, but should perform better on the new version of REBOL. RG> 1. Every database has a leading "Master" block. This block [...] RG> I am leaving out lots of details for brevity and at this RG> point want to follow Gabriele's lead and stop here and wait RG> for reactions. I don't want to run off in a direction no one ...I've got a nice idea. :) After I have finished the on memory low level functions, could you try doing a compatible version that uses your binary format? So we'll offer four alternatives: in memory, on disk a file per block (text), on disk one file (text), on disk binary. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [22/50] from: g:santilli:tiscalinet:it at: 14-Jan-2002 22:00


Hello Romano! On 14-Gen-02, you wrote: RT> 1) I like to have a function to update the db. Until the RT> funtions call all the changes should resides in memory or in RT> a temp file. I didn't want to use any cache in dbms.r to minimize data loss on crashes. Maybe we could use memory tables in dbms3.r for something like that. RT> 2) I do not like to use a dir for a db. Neither to use a dir RT> name for a db name. My preference is for basename. Yes... when I started it i thought: "well, let's start with a simple file stucture, and extend later". (I wanted it for a mailer, so not much rows per table.) I never got around to extend it... I'll do it now with version 3. :) RT> 3) I should like a RT> db-select dbname none It's equivalent to: load dbname/index RT> 4) I do not like 1 record 1 file: on Windows cluster are of RT> 32-64kb (what a beatiful thing!) but i know changing this is RT> your actual first target. Indeed, same reason. RT> 5) Do you keep in memory Columns and indexes? Nope, that was one of the things "I'll do it later"... :) Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [23/50] from: g:santilli:tiscalinet:it at: 14-Jan-2002 22:48


Hello Romano! On 14-Gen-02, you wrote: RT> Now i realize that my Rebol databases are made of nested RT> blocks of variable length. It is not clear to me how to RT> convert them in a row/column scheme. Any ideas? The process is usually called normalization. :) If you can provide an example, maybe I can try to give a possible solution. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [24/50] from: g:santilli:tiscalinet:it at: 14-Jan-2002 22:45


Hello Romano! On 14-Gen-02, you wrote: RT> I can do it with db-foreach, but i think is slow to convert RT> data in object and then in block. I feel that block are the Earlier versions of dbms.r used blocks instead of objects, but I found that not practical. It's not useful to have to remember if the row you need (e.g. 'name) is the second or the third in the block... :) RT> Perhaps could exist two series of interfaces: blocks and Maybe... RT> (i say everything i can wish/think, hope not to exaggerate) Feel free to! :) Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [25/50] from: rotenca:telvia:it at: 15-Jan-2002 1:52


Hi Rod, Gregg, Gabriele
> The process is usually called normalization. :) If you can provide > an example, maybe I can try to give a possible solution.
Now i realize how it is possible (i made it tens of times in databases programs :-). But i feel it ab-normal. I must extract data from a "natural" container to put it in an artificial one which can be useful only in some rare cases and I must duplicate at least one value to keep the link with the "master" db and add a ID for every record. It is the prize of relational row/column struct. It has the advantage of a direct access to the data of the derived db. I'm starting ask if it is the best path to follow. --- Ciao Romano

 [26/50] from: rotenca:telvia:it at: 15-Jan-2002 1:59


Hi Gabriele
> I didn't want to use any cache in dbms.r to minimize data loss on > crashes. Maybe we could use memory tables in dbms3.r for something > like that.
I prefer to loose a session of add/remove/change than to mess a whole db.
> RT> db-select dbname none > > It's equivalent to: > > load dbname/index
I did not try, but what i want is a unsorted collection of all column of all records.
> RT> 5) Do you keep in memory Columns and indexes? > > Nope, that was one of the things "I'll do it later"... :)
I hope you will use Hash! --- Ciao Romano

 [27/50] from: rgaither:triad:rr at: 14-Jan-2002 19:18


Hi Gabriele, I got your other response to my questions etc. All good points that explain what you need well so I didn't reply to it. :-)
> RG> I am assuming (dangerous I know) that binary storage is an > RG> option at all. If you want a text based storage system then > RG> ignore this input. :-) > >Text based for simplicity, but... (see below)
Ok then - text based for this project. Do you have a file format in mind yet? I would be more comfortable thinking about functions if I could visualize how you are thinking about storing the data. If we can come up with a standard, REBOL friendly way of creating a relational text DB file format then people are free to use whatever tools they want to read and write the files. The key value comes in the open, standard format definition not the specific tools. Some design thoughts - 1. I would prefer a single file DB if possible. It doesn't rely on a directory structure and includes everything it needs in one simple, transportable, format. 2. I would also like to store the REBOL values in their native text format. 3. The database should support Tables, Rows, Columns, Indexes, and Named Sequences for starters with possibly a special field type that uses a Sequence to create auto-incrementing ID values to simplify creating primary keys. I'm going to play around with some format options and include them in another post.
> RG> I am concerned in my ignorance about your comment on REBOL > RG> not being able to seek "easily" on direct ports. Could you
<<quoted lines omitted: 3>>
>you need to go back... This is expected to change in the next >release.
Thanks a bunch for the explanation! I was not aware of this limitation.
>...I've got a nice idea. :) After I have finished the on memory >low level functions, could you try doing a compatible version that >uses your binary format? So we'll offer four alternatives: in >memory, on disk a file per block (text), on disk one file (text), >on disk binary.
Yes I can. Trouble is that would mean I'd have to have the binary storage version working first. :-) Still, it is a good plan and I will try. I do want to help with the text version as well though. I have several simple DB needs that it could solve very nicely. I will review dbms.r so I understand what you have already done better. Thanks! Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [28/50] from: rgaither:triad:rr at: 14-Jan-2002 20:23


Hi Romano,
>> The process is usually called normalization. :) If you can provide >> an example, maybe I can try to give a possible solution.
<<quoted lines omitted: 6>>
>has the advantage of a direct access to the data of the derived db. I'm >starting ask if it is the best path to follow.
The model where the data gets its relationship(s) from its "nested" position was known as "Hierarchical" if I remember it correctly. :-) This was the leading model when relational came on the scene. In general the relational model had the advantage of flexibility in dealing with more complex data relationships. It was however at the price you mention of duplicating and providing the ID information which can often be more columns than the real data in a fully normalized database. As for the best path to follow it "depends". :-) It is worth noting that XML is breathing some life back into hierarchical representation of data. FWIW, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [29/50] from: rotenca:telvia:it at: 15-Jan-2002 2:53


Hi Gabriele,
> Earlier versions of dbms.r used blocks instead of objects, but I > found that not practical. It's not useful to have to remember if > the row you need (e.g. 'name) is the second or the third in the > block... :)
Yes but often we need blocks of blocks of data. There at least three methods to not have to remember positions (you know them): column: [name surname email] data: ["joe" "smith" [joe--smith] ] 1) set column data probe name 2) column: reduce ['name pick data 1 'surname pick data 2 'email pick data 3] probe column/name 3) transform the block in an object I think that is the user which must decide it. The database should offer only a collection (block) of data not a fixed struct like an object which cannot be extedend without cloning (and the definition of which, it seems to me, is created by your program at each iteration) and that is not useful with any of Rebol function which all ask blocks to share data (except a few cases like face, event, error ...). To find replace sort a collection of objects is not a direct task. A block can be easy converted in another datatype! like an hash! for fast search. I think also that the layout of object (if it exists) could be defined by user and dbms should only bind data to the user object. The object interface should be an option, an additional service. Object interface is surely useful for db-foreach, but i always think at a kind of db-select [name surname] [sort by name] which return a block of blocks. --- Ciao Romano

 [30/50] from: petr:krenzelok:trz:cz at: 15-Jan-2002 6:26


Hi, Maybe the block structure is the way to go. If you look at current ODBC Rebol solution, it retrieves block of blocks (records), transformed or not, into Rebol datatypes. I already have several functions, where I can map such record to object - very comfort aproach, using foreach .... foreach record record-set [rec: map-2-object template record .... do something with that object ...] , where template can be retrieved as simply another query to datase (in case of ODBC "Insert odbc-port ['columns dbname]), or just predefined (then only fields available in template map ....) What I am more interested in is - data integrity. If we will use only one file for our db, do we have to save upon each change? Because if we don't do that, and our app crashes, all changes are lost .... As for unique IDs, Elan took the right aproach imo. Rebol is about distribution, so don't start creating dumb system, if you really don't require only that, and nothing more. But I know the tastes - once you finish it, you will want to add another features. Elan for e.g. created system, where each video store could be assigned unique ID thru CGI process on some remote machine - ID unique to all video stores using the app. Also - per-file-record was MUCH better aproach, than storing everything in one file, unless you open the file on disk and forget memory based aproach. What is a little bit of a problem is - you use file system, so performance can vary. But hey, you just wanted to handle some 100 - 10K records, no? Again - look at what Elan did, and improve upon it, IIRC he used some techniques, which could be improved .... As for database normalisation - come on, guys ;-) You want to achieve some simple db functionality, yet considering db normalisation? We created some middle size system (cca 70 tables, some of them having > 100 columns and 200K records), yet we sometimes use email1 email2 just in one record, not providing sub-related table. As for fields duplication in sub-related (daughter) tables - yes, it's like that, but no real problem here .... As for me, once open/seek mode is available, I would try to go with XBase format - many tools use it - FoxPro, Clipper, Visual Objects, Visual Basic, Delphi can, etc. etc. - there are already tools to browse such data, and I can provide you with format description. But, huh, too much of work and wheel reinvention in any case :-) So, for me, even if done in Rebol, I can't think of anything simpler then what Elan created, although I don't like the idea of some tens of thousands of records scattered around my hd in form of small files :-) -pekr-

 [31/50] from: brett:codeconscious at: 15-Jan-2002 19:11


With so much good thinking on this topic I'm a little intimidated to make my comment! But anyway ... Scanning the posts I didn't see anyone point out that an in-memory structure can look very different from an on-disk structure. Maybe it is too obvious. If someone said it - sorry for the repeat! So you can optimise your in-memory structure to handle the expected access requests in an efficient manner. The out-of-memory (persistent) structure could be more attuned to other requirements or perhaps there could be multiple formats. The trick will be in binding the two together. I refer to it as an out-of-memory structure rather than an on-disk structure because you may want to bind it not just with a disk file but perhaps with some custom xml parser, or mime-email or port, or. whatever. Gabriele, if you achieve everything I've seen on the list it will be the best little database ever! :) Regards, Brett.

 [32/50] from: robert:muench:robertmuench at: 15-Jan-2002 11:19

Re: [DEV] Re: dbms3.r 01


> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]On Behalf Of > Gregg Irwin > Sent: Monday, January 14, 2002 9:13 PM > To: [rebol-list--rebol--com] > Subject: [REBOL] Re: [DEV] Re: dbms3.r 01
Hi, just some comments from my POV. IMO a tiny, simple to use persistence system for Rebol is a very good thing and I could use it in almost all everyday situations. Up to now, several ways-to-do-it exist and it would help me a lot if a best-practice shows up. My requirements would be: - Simple file format, so I can use other scripts to work on them and do some analyze things, without having to use the database-driver. - Simple multiple-user support. Exclusive object-locking would be good. - Some form of simplest transaction handling. Just to be sure not to get into a instable datastructure state. - Close relationship to Rebol's datastructure block! and object!
> I want to do as little work as possible, at the application level, to store > and retrieve my data. I love the idea of a transparent persistent object > system, with the ability to define relationships not implied by an object > hierarchy, ...
And yes, that's what I like to.
> A few years ago, I saw what I thought was a very well designed ORDBMS > (orbject-relational) product. It was called Cascade/DB, from Microquill.
I know this one. Unfortunately I never got a version to test it and didn't had a chance to understand the concept better.
> Another company bought the product and it vanished from sight (and > Microquill couldn't name the company who bought it, as part of the deal).
IMO is could have been MS... as these features poped-up in SQL server, IIRC.
> So, for me, I want a simple system that is tightly focused on storing and > retrieving REBOL data (probably simple blocks, blocks of name/value pairs, > and objects). It should handle up to maybe 50,000 records, though most cases > will be far less I imagine. The little things I've done so far have actually > worked well using native storage, but the scalability, robustness, and > multi-user aspect are something I'm concerned about for more critical uses.
Yep, same for me! And here is a suggestion: Why do we want to implement it in Rebol at all? How about a real simple data-storage system that can be used through ports from Rebol, that knows about Rebol's datatypes, etc.? IMO we should write a portable rebol-focused-simple-data-storage-system that can be used through ports. All the little things you have requested are much easier to write in C/C++, or something else and get it connected to Rebol. I want to use Rebol for GUI and process design but not for the requested data-storage. With this approach, I would go for a non-relational-model. More a graph based one, that supports lists in nodes etc. Like Cascade allowed it. Do some of you know Xandadu (http://www.xanadu.net)? It has a concept of a multi-dimension data-storage, maybe there are some ideas we can use. I really would be interested in developing such a system, because I need one for my projects too ;-)). -- Robert M. Münch IT & Management Freelancer Mobile: +49 (0)177 2452 802 Fax : +49 (0)721 8408 9112 Web : http://www.robertmuench.de

 [33/50] from: petr:krenzelok:trz:cz at: 15-Jan-2002 12:13


> Yep, same for me! And here is a suggestion: Why do we want to implement it in > Rebol at all? How about a real simple data-storage system that can be used
<<quoted lines omitted: 3>>
> to write in C/C++, or something else and get it connected to Rebol. I want to > use Rebol for GUI and process design but not for the requested data-storage.
well, I can imagine it, but there is one significant limitation to overcome - you have to launch such external database gateway in some way. Unless /Shell component comes for free, some ppl will not regard the solution being pure Rebol based .... On the other hand - look at Rebol ODBC solution for e.g. On one hand you use simplified port based mechanism, while on the other hand you need to understand external language, e.g. some SQL equivalent. So - you would have to include database commands into your C/C++ engine, which could limit most users, because changes are difficult to introduce with such solution ... But if you want to do it, look at "multiplatform" LibmySQLd, create the gw app, and you are done .... well, binary format though, which is not suitable from what you described ... -pekr-

 [34/50] from: rgaither:triad:rr at: 15-Jan-2002 8:55

Re: dbms3.r 01


Hi Pekr,
>Maybe the block structure is the way to go. If you look at current ODBC Rebol >solution, it retrieves block of blocks (records), transformed or not, into Rebol >datatypes. I already have several functions, where I can map such record to >object - very comfort aproach, using foreach ....
I think a block structure is the way to go at the storage level. I think the tools/driver will need to be objects. A result set could be returned as a block though.
>foreach record record-set [rec: map-2-object template record .... do something >with that object ...] > >, where template can be retrieved as simply another query to datase (in case of >ODBC "Insert odbc-port ['columns dbname]), or just predefined (then only fields >available in template map ....)
Yes, this is a good relational point - the schema or template stored and operated on in the same table/row/column structures.
>What I am more interested in is - data integrity. If we will use only one file for >our db, do we have to save upon each change? Because if we don't do that, and our >app crashes, all changes are lost ....
These issues exist for any number of files used. It is something to think about for any of the format options.
>As for unique IDs, Elan took the right aproach imo. Rebol is about distribution, >so don't start creating dumb system, if you really don't require only that, and >nothing more. But I know the tastes - once you finish it, you will want to add >another features. Elan for e.g. created system, where each video store could be >assigned unique ID thru CGI process on some remote machine - ID unique to all >video stores using the app.
I don't like the remote machine approach (ID server or service). I would prefer something along the lines of namespaces and local ID management.
>Also - per-file-record was MUCH better aproach, than storing everything in one >file, unless you open the file on disk and forget memory based aproach. What is a >little bit of a problem is - you use file system, so performance can vary. But >hey, you just wanted to handle some 100 - 10K records, no? Again - look at what >Elan did, and improve upon it, IIRC he used some techniques, which could be >improved ....
I will review what Elan did in his book but I don't remember being comfortable with it the first time around. Again I think it had directory requirements as well as too many files for my preferences.
>As for database normalisation - come on, guys ;-) You want to achieve some simple >db functionality, yet considering db normalisation? We created some middle size
Guilty as charged. :-) I want multi-user and transaction support as well...
>As for me, once open/seek mode is available, I would try to go with XBase format - >many tools use it - FoxPro, Clipper, Visual Objects, Visual Basic, Delphi can, >etc. etc. - there are already tools to browse such data, and I can provide you >with format description. But, huh, too much of work and wheel reinvention in any >case :-)
It is tempting but not what I am after. If we get to the binary version of this effort that format will not be what I pick. Might as well use a DLL from Delphi and work through it at that point.
>So, for me, even if done in Rebol, I can't think of anything simpler then what >Elan created, although I don't like the idea of some tens of thousands of records >scattered around my hd in form of small files :-)
That is the challenge of it isn't it! :-) Thanks, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [35/50] from: robert:muench:robertmuench at: 15-Jan-2002 15:12

Re: [DEV] Re: dbms3.r 01


> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]On Behalf
<<quoted lines omitted: 3>>
> Subject: [REBOL] Re: [DEV] Re: dbms3.r 01 > well, I can imagine it, but there is one significant limitation to overcome -
you
> have to launch such external database gateway in some way. Unless > /Shell component comes for free, some ppl will not regard the solution being
pure
> Rebol based ....
Hi, that's right but with a startup-script it should be OK.
> On the other hand - look at Rebol ODBC solution for e.g. On one hand you use > simplified port based mechanism, while on the other hand you need to > understand external language, e.g. some SQL equivalent. So - you would have to > include database commands into your C/C++ engine, which could limit most
users,
> because changes are difficult to introduce with such solution ...
Not quite. For Rebol you would get a dialect for database activity, queries etc. The engine is just for doing the hard work. I don't want to implement a SQL parser there ;-)). The engine can search on b*-trees etc. even the query optimizer can be done in Rebol if you want.
> But if you want to do it, look at "multiplatform" LibmySQLd, create > the gw app, and you are done .... well, binary format though, which is not
suitable
> from what you described ...
This is a RDBMS... I don't want to use a RDBMS concept, the engine should be much more Rebol coupled. Robert

 [36/50] from: rgaither:triad:rr at: 15-Jan-2002 9:28

Re: dbms3.r 01


Hi Brett,
>With so much good thinking on this topic I'm a little intimidated to make my >comment! But anyway ... > >Scanning the posts I didn't see anyone point out that an in-memory structure >can look very different from an on-disk structure. Maybe it is too obvious. >If someone said it - sorry for the repeat!
No, it is a good point to make. There is value in having them the same in simplicity for the functions but there is also value in not being limited by the serial nature of the persistent text file. It certainly should be explored the same way the text file options are being explored.
>So you can optimise your in-memory structure to handle the expected access >requests in an efficient manner. The out-of-memory (persistent) structure
<<quoted lines omitted: 3>>
>because you may want to bind it not just with a disk file but perhaps with >some custom xml parser, or mime-email or port, or. whatever.
Both are good points!
>Gabriele, if you achieve everything I've seen on the list it will be the >best little database ever! :)
Hah, hopefully Gabriele will pick a couple key options and go about creating something instead of letting us design and design and design... Sometimes it is good to have deadlines! :-) Thanks, Rod. Rod Gaither Oak Ridge, NC - USA [rgaither--triad--rr--com]

 [37/50] from: jason:cunliffe:verizon at: 15-Jan-2002 1:17


Have you guys taken a look at 'kd'b by kx systems? http://www.kx.com/ http://www.kx.com/download/download.htm ...it is lightning fast and tiny Tens of millions of records--stored or streamed--read in a second. Fast relational database (SQL92 with ODBC and JDBC), extended for time-series queries. The Kdb Viewer is a program for accessing, viewing, creating and analyzing data. There is also a web Viewer. A local version of the web Viewer comes with download Lots of REBOL like aspects: tiny smart different Well worth playing around with I believe while you are at this stage. ./Jason

 [38/50] from: greggirwin:mindspring at: 15-Jan-2002 11:50

Re: [DEV] Re: dbms3.r 01


Hi Robert, et al << And here is a suggestion: Why do we want to implement it in Rebol at all? How about a real simple data-storage system that can be used through ports from Rebol, that knows about Rebol's datatypes, etc.? >> I think it *absolutely* should be implemented in REBOL, for the following reasons: 1) It will be easier! If it is implemented in REBOL and then someone wants to try and improve performance, that's fine, but I think it will be a *lot* more work to do it in C. 2) It will serve as a terrific example of what you can do with REBOL. 3) You will get a lot more eyes on it, that understand the code, from this ML. I can read C if I have to, but I don't like it. I like REBOL. 4) It will be more portable with less effort. 5) If/when RT adds new datatypes, if we haven't gotten tricky, it should automatically support them. If not automatically, we should consider how to easily extend it for that eventuality. Just my opinion. We seem to agree far more than we disagree. :) --Gregg

 [39/50] from: g:santilli:tiscalinet:it at: 17-Jan-2002 20:56

Re: dbms3.r 01


Hello Romano! On 15-Gen-02, you wrote: RT> But i feel it ab-normal. I must extract data from a "natural" RT> container to put it in an artificial one which can be useful [...] Of course it depends on your data and your application! Usually I use the relational model when I design the data structure with the ER model etc. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [40/50] from: g:santilli:tiscalinet:it at: 17-Jan-2002 21:12


Hello Romano! On 15-Gen-02, you wrote: RT> I did not try, but what i want is a unsorted collection of RT> all column of all records. DB-SELECT only gives you the row-ids; getting all the records in a block is not possible with dbms.r, as I've never had a need for that; to get them as a block of objects, you can use: block: make block! 1024 db-foreach row %table/ none [ insert tail block make row [] ] The MAKE is needed because DB-FOREACH reuses the same object across evaluations for efficiency. RT> I hope you will use Hash! I was thinking about b-trees... but hash! is certainly an option. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [41/50] from: g:santilli:tiscalinet:it at: 17-Jan-2002 20:49


I'v been away for a few days, now I have to play catch up and prepare part two of dbms3.r dev. :) Hello Rod! On 15-Gen-02, you wrote: RG> Do you have a file format in mind yet? I would be more RG> comfortable thinking about functions if I could visualize how RG> you are thinking about storing the data. REBOL's native format is my choice, both for simplicity and to allow for dynamic sized records easily. To allow random access to the table file, we'll need an index of the records, that can be kept as a block of integers. RG> If we can come up with a standard, REBOL friendly way of RG> creating a relational text DB file format then people are RG> free to use whatever tools they want to read and write the RG> files. The key value comes in the open, standard format RG> definition not the specific tools. Hmmm... let's wait for what I come up with when I'll do the one-file table functions; then we can discuss how to improve it etc. RG> 1. I would prefer a single file DB if possible. It doesn't RG> rely on a directory structure and includes everything it RG> needs in one simple, transportable, format. I'd like to keep indexes separately anyway, but then again, we can discuss this later. RG> 2. I would also like to store the REBOL values in their RG> native text format. Yup! RG> auto-incrementing ID values to simplify creating primary RG> keys. That's a useful feature, so it's very likely it will be there. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [42/50] from: g:santilli:tiscalinet:it at: 17-Jan-2002 21:22


Hello Romano! On 15-Gen-02, you wrote: RT> Yes but often we need blocks of blocks of data. Actually I think my code got a lot better when switching from the old block-based dbms.r to the new object-based one... RT> should offer only a collection (block) of data not a fixed RT> struct like an object which cannot be extedend without Actually the relational model includes a name for the column; other languages (PHP etc.) offer records as associative arrays, and the thing that most resembles associative arrays in REBOL is object!. RT> cloning (and the definition of which, it seems to me, is RT> created by your program at each iteration) and that is not DB-FOREACH reuses the same object across evaluations for efficiency. Of course DB-PICK can't do something like that, so it will create a new object each time. RT> To find replace sort a collection of objects is not a direct RT> task. Indeed that's something that the DBMS should do, not you. REBOL does not have a table! datatype... :) RT> A block can be easy converted in another datatype! like an RT> hash! for fast search. You'll need to search in a record? That's quite strange IMHO. :) Better redesign your table, but you can use SECOND OBJECT anyway if you need. RT> I think also that the layout of object (if it exists) could RT> be defined by user and dbms should only bind data to the user It is, when the user defined the table. RT> Object interface is surely useful for db-foreach, but i RT> always think at a kind of db-select [name surname] [sort by RT> name] which return a block of blocks. I'm not sure about its usefulness, but I might include something like that in dbms3.r. Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [43/50] from: rotenca:telvia:it at: 18-Jan-2002 21:03


Hi, Gabriele
> RT> Yes but often we need blocks of blocks of data. > > Actually I think my code got a lot better when switching from the > old block-based dbms.r to the new object-based one...
Well. You like object, i like blocks. When i'll have the basic i/o routines (the hard part) i'll write my own block select routines. There is no problem at all.
> Actually the relational model includes a name for the column; > other languages (PHP etc.) offer records as associative arrays, > and the thing that most resembles associative arrays in REBOL is > object!.
Ok. But i want only duplicate a SQL SELECT FROM ... ORDER BY .... I do not neither more. I don't want to write a DB interface programs i want only to use data for programs which make a specialized things. I feel myself a true client :-). I do not want to see the structure of data. I do not care about it.
> RT> To find replace sort a collection of objects is not a direct > RT> task. > > Indeed that's something that the DBMS should do, not you. REBOL > does not have a table! datatype... :)
No. I want to make it myself. I repeat: the db for me is only "a standard way" to store data. When i write a program often i have to store/retrieve data. In the past i have written custom routines for the task. I think it is better to have a set of routines/format to do the task in all my programs.
> RT> A block can be easy converted in another datatype! like an > RT> hash! for fast search. > > You'll need to search in a record? That's quite strange IMHO. :) > Better redesign your table, but you can use SECOND OBJECT anyway > if you need.
The block is a block of blocks, block of selected field of records.
> RT> Object interface is surely useful for db-foreach, but i > RT> always think at a kind of db-select [name surname] [sort by > RT> name] which return a block of blocks. > > I'm not sure about its usefulness, but I might include something > like that in dbms3.r.
I use it for list and text-list, my main need. But when there are the basic i/o functions is easy to write it by myself. 9/10 of my code does a query on a data set and visualize it, then the user select something and start an action which it is not related to the database itself in many cases. --- Ciao Romano

 [44/50] from: greggirwin:mindspring at: 18-Jan-2002 14:23


Hi Gabriele, << Actually the relational model includes a name for the column; other languages (PHP etc.) offer records as associative arrays, and the thing that most resembles associative arrays in REBOL is object!. >> I think I would say that a block of word/value pairs (e.g. [a 1 b 2]) is more like an associative array. Can an object's spec block be extended, or modified, after it is created? --Gregg

 [45/50] from: robert:muench:robertmuench at: 19-Jan-2002 11:28


> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]On Behalf Of
<<quoted lines omitted: 4>>
> Can an object's spec block be extended, or > modified, after it is created?
Hi, yes it can: extend-object: func ['obj-word [word!] 'word [word!] value [any-type!] /local the-obj][ all [not object? the-obj: get obj-word make error! "No object provided."] if in the-obj word [ set in the-obj word value return the-obj ] set obj-word make the-obj reduce [to-set-word word value] ] -- Robert M. Münch IT & Management Freelancer Mobile: +49 (0)177 2452 802 Fax : +49 (0)721 8408 9112 Web : http://www.robertmuench.de

 [46/50] from: rotenca:telvia:it at: 19-Jan-2002 14:18


Hi Robert,
> > Can an object's spec block be extended, or > > modified, after it is created? > > Hi, yes it can: > > extend-object: func ['obj-word [word!] 'word [word!] value [any-type!]
/local
> the-obj][ > all [not object? the-obj: get obj-word make error! "No object provided."]
<<quoted lines omitted: 4>>
> set obj-word make the-obj reduce [to-set-word word value] > ]
No, it can't. What your rotine do is to create a new object:
>> x: x1: context [a: 1] >> y: extend-object x b 2 >> probe x1
make object! [ a: 1 ] --- Ciao Romano

 [47/50] from: g:santilli:tiscalinet:it at: 19-Jan-2002 12:12


Hello Romano! On 18-Gen-02, you wrote: RT> Well. You like object, i like blocks. When i'll have the RT> basic i/o routines (the hard part) i'll write my own block RT> select routines. There is no problem at all. That's fine. :) I'll support you in doing that. RT> 9/10 of my code does a query on a data set and visualize it, I do that too. :) Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [48/50] from: g:santilli:tiscalinet:it at: 19-Jan-2002 17:23


Hello Gregg! On 18-Gen-02, you wrote: GI> I think I would say that a block of word/value pairs (e.g. [a GI> 1 b 2]) is more like an associative array. Can an object's GI> spec block be extended, or modified, after it is created? It can't, so you are right; but an object still seems the best solution in my case. Remember that: b: [a 1 b 2] b/b ; is O(n) while contexts are likely to be optimized for word lookups. (Well, I know Ladislav has something to say about this...) Regards, Gabriele. -- Gabriele Santilli <[giesse--writeme--com]> - Amigan - REBOL programmer Amiga Group Italia sez. L'Aquila -- http://www.amyresource.it/AGI/

 [49/50] from: chalz:earthlink at: 21-Jan-2002 0:28


Okay, here's another comment from someone who hasn't really worked with this material. I believe someone else's solution was to, essentially, create a new object. What about that? Sort of like children/evolution - if something like that needs to be changed, couldn't you instead create a new copy of ObjA with the additional info, then erase ObjA and replace it with ObjA.1?

 [50/50] from: greggirwin:mindspring at: 21-Jan-2002 10:04


Hi Charles, << Okay, here's another comment from someone who hasn't really worked with this material. I believe someone else's solution was to, essentially, create a new object. What about that? Sort of like children/evolution - if something like that needs to be changed, couldn't you instead create a new copy of ObjA with the additional info, then erase ObjA and replace it with ObjA.1? >> Sure you can. Object versioning and schema evolution are two features that some object databases tout as major benefits. The trick is how to do it correctly. :) Some example questions might be: If you have an object in memory, and other code holds references to that object, what happens to those references when you destroy, and recreate, the object? Do you re-use the same object ID, or do you assign a new object ID so that the old version of the object can still be accessed (or do you assign a "versioned" ID to the old one)? Do you make a distinction between a schema change and a data change? Things like that. The Devil is in the details as they say. --Gregg

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted