Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

NTFS Stream Fun

 [1/23] from: robert::muench::robertmuench::de at: 5-Aug-2002 17:40


Hi, I don't know if someone has posted something about this already. Anyway, ever heard of NTFS streams? This is in fact a way to store multiple data-tacks in one file. And MS has a very simple usage pattern (at least that's what they sometimes get right) that even works with Rebol!! Have a look:
>> write %test.txt "This is a test." >> print read %test.txt
This is a test.
>> write %"test.txt:hidden" "This is an other test stored in a stream." >> print read %test.txt
This is a test.
>> print read %"test.txt:hidden"
This is an other test stored in a stream. Cool, eh? Have a look at the file "test.txt", you will only see the 15 bytes from the first write. The stream named "hidden" can't be detected with just normal tools. Of course this only works on NTFS filesystem drives. But I think it's a real cool thing. Imagine what you could do with this: You can add your library files into a "library" stream and have them loaded at runtime. No need to pack scripts etc. Robert

 [2/23] from: ptretter:charter at: 5-Aug-2002 17:09


Very cool Robert. Good info to know. Paul Tretter

 [3/23] from: atruter:hih:au at: 6-Aug-2002 10:42


Wow! This could be a good way to organise data files. Instead of having pcodes-vic.dat, pcodes-nsw.dat, etc have pcodes.dat:vic, pcodes.dat:nsw. Makes me wonder if this could be useful for DB type apps . . . db-file:key? Think I'll see if there are any documented limits / performance issues with this "feature" (sounds similar to resource forks under Mac). Thanx for sharing this find . . . Regards, Ashley

 [4/23] from: bga:bug-br at: 5-Aug-2002 22:09


> Wow! This could be a good way to organise data files. Instead of > having
<<quoted lines omitted: 5>>
> issues with > this "feature" (sounds similar to resource forks under Mac).
It is not directly related to Rebol, but I would like to point out that BFS, the native filesystem under BeOS does support metadata (attributes) and the number of them per file is unlimited. Also the size of each of them is unlimited. Attributes of integral types (int, string) can be indexed automatically and transparently by the FS and you can them search files in a way you would do in a database ("show me all mpeg audio files that have Rock as the Genre, that has been composed after 1990 and that I gave a Rating of 10 to it"), all that without no need for any specific app (using just the system Find tool). Not to mention queries are live and the query result is updated on the fly if you add/remove files that match the criteria. In other words, it completelly smokes NTFS out. :) -Bruno -- Fortune Cookie Says: Have you noticed that all you need to grow healthy, vigorous grass is a crack in your sidewalk?

 [5/23] from: gschwarz:netconnect:au at: 6-Aug-2002 11:27


Ashley, I noticed your e-mail address. Is hih address going to be around for a bit longer? Regards, Greg

 [6/23] from: atruter:hih:au at: 6-Aug-2002 12:09


> I noticed your e-mail address. Is hih address going to be around for a
bit longer? <off topic> I finish up end of October, so expect a different address from me shortly after that. Last folks to use "hih.com.au" will probably be off the old mail servers by March. Company will then take another decade to liquidate totally (long tail claims, court actions, creditor payments, etc). Most folks don't realise that a financial institution (bank, insurer, etc) rarely just shuts up shop - there are many many financial, accounting and legal issues of mid to long term duration that need to be resolved. </off topic> Regards, Ashley Truter (contracted to HIH in liquidation)

 [7/23] from: robert:muench:robertmuench at: 6-Aug-2002 11:05


> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]
<<quoted lines omitted: 4>>
> It is not directly related to Rebol, but I would like to > point out that BFS, the native filesystem under BeOS does support
metadata
> (attributes) and the number of them per file is unlimited.
Hi, yes the BFS is a nice thing. I liked it a lot. NTFS supports this too, even it's mostly unknown. Ever looked at the "Summary" tab of a file properties dialog? All the fields you see there is meta-data stored with the file. This data is stored in an internal known steam. However, IIRC you can have a lot of streams in a NTFS file. BTW: The stream feature is AFAIK available since NT 3.1 (1994). What you see in a directory listing is the so called "default" stream. This stream has the form of <filename>:$DATA Further NTFS streams are not normally visible - unless you know one is there (and unless you know its name), you are out of luck. If you copy a multi-stream file to a FAT volume, all the streams will disappear except the unnamed stream. In addition, applications that transfer files and are not stream-aware only transfer the main unnamed stream, for example email clients. This makes it useful for things like license keys etc. Of course you can have a look at the MFT of a NTFS partition but all these steps need a lot of knowledge about NTFS. And Microsoft uses streams mostly for Macintosh support. Whereas the Windows 2000 Content Indexing Server adds an alternate data stream named "?Q30lsldxJoudresxAssqpcawXc" to image files on NTFS volumes. This stream contains thumbnails of the images. An even less common application for NTFS streams is the creation of hard links (Posix-style file links). These get done by writing, to a special stream ID (WIN32_STREAM_ID/BACKUP_LINK), the full path of the new link to create. And streams can not only attach themselves to files, they can also attach themselves to directories. This is a not so nice feature as streams can't actually be deleted. The parent they're attached to must be deleted in order for the stream to be removed. If you attach a stream to the root directory of a drive, such as "C::MyStream", you can't delete this one at all. So be careful. Further streams, as they are essentially still files, can be executed and do not have their filenames display correctly in Windows NT/2K Task Manager, the utility commonly used to view running processes. For example, if the stream "c:\test.txt:mystream" was running, Task Manager would only show "test.txt". And while we are at it, try to find a web-site running some older IIS servers and type in http://www.victim.com/target.asp::$DATA You will see the source-code of the ASP file. The server won't execute the code. However, operating systems can access streams on an NTFS share, independent of their file system. For example, a Windows 95 operating system with a stream aware client might view a stream on a Windows NT 4.0 NTFS network share.
> In other words, it completelly smokes NTFS out. :)
NTFS is not that bad. I like some features of BFS as well but IMO the NTFS is really a nice thing. I don't remember the title of a book (if you know it let me know it too) that tells the story about the NT development. Very good read! Not very technical, instead it tells about the people and up & downs while developing NT. There is a section about the development efforts that went into NTFS etc. Robert

 [8/23] from: bga:bug-br at: 6-Aug-2002 13:38


On Wed, 7 Aug 2002, Anton wrote:
> This is the first of streams that I have heard. > I am sceptical about its long-term value, except to
<<quoted lines omitted: 12>>
> translate between the two namespaces - a waste of time, > in my view.
If you move a file, the metadata follows it. if you detete a file, the metadata is automatically deleted. Metadata (or extended attributes) are used to store information that is relatd to the file but is not part ofthe file itself. For instance you have a plain text file and you store the Author's name in a extended-attribute so you can have this information available without having to edit the contents of the file. Image thumbnail is another example. BeOS comes with a text editor (StyledEdit) that lets you write text files using simple text formating (changing fonts, colors and font size of parts of the text) but after you save it you can open the file in an standard text editor and although you will not see the special formatting you did, there wll be no garbage in the file, only the text itself. All formating information is saved as an attribute. -Bruno

 [9/23] from: anton:lexicon at: 7-Aug-2002 1:59


This is the first of streams that I have heard. I am sceptical about its long-term value, except to support applications that use it already (like Mac forks). Isn't it just creating a new namespace - therefore creating more work in creating/accessing/deleting the streams? So, you can hide some streams in one file, keeping them nicely together. But what is the advantage over a set of files in a directory? We already know how to create, access, move, delete, compress, decompress etc. files in a filesystem, why create another namespace? Then we just have to write a set of functions that translate between the two namespaces - a waste of time, in my view. But perhaps there is a high-performance situation where this NTFS feature is really useful. If someone can point it out, then I might change my mind. :) Anton.

 [10/23] from: kemp:extelligence at: 6-Aug-2002 13:15


Streams are part of Windows/NTFS Structured Storage, and are used to support a number of specialized file formats. For example, Kodak's Flashpix image format uses structured storage streams to hold thumbnails, full images, and various mangifications in-between, with each level of resultion being tiled into it's own stream. On top of that, it stores standard and custom metadata in other streams. All of this is related and inseparable, and streams allow it to go nicely into one file without having to parse through stuff you don't always need or shouldn't see. - Kemp

 [11/23] from: atruter:hih:au at: 7-Aug-2002 10:31


> what is the advantage over a set of files in a directory?
Data integrity. Regards, Ashley

 [12/23] from: anton:lexicon at: 7-Aug-2002 14:30


So you're saying that the fact that the streams are more difficult to modify & and access is what makes them useful. A bit like information-hiding and public interfaces from object orientation. Seems like a useful concept - I can buy that. Although, what happens when you want to get rid of some "meta-data"? Or copy some meta-data to another file? It seems much like an archive, containing a bunch of files, one of which is a "default" file that has the data you probably want. If archives were transparently incorporated into the filesystem, then I can imagine accessing their contents directly using path notation: example.zip/readme.txt but referring to: example.zip would be equivalent to: example.zip/default.binary File streaming was obviously designed to solve some specific problems, so I am not criticising it as such, just the excited reaction as if it is some wonderful new way of expressing relationships between clumps of data. File streams just add some special leaves to the branches of the hierarchical filesystem tree. Anton.

 [13/23] from: atruter:hih:au at: 7-Aug-2002 15:44


> File streaming was obviously designed to solve some > specific problems, so I am not criticising it > as such, just the excited reaction as if it is > some wonderful new way of expressing relationships > between clumps of data.
I am probably one of those who have an "excited reaction" and for me this is a "wonderful new way of expressing relationships between clumps of data"; not necessarily *conceptually new* but new in terms of what I now know NTFS is capable of, and the fact that REBOL transparently handles this. No I haven't used this yet and I don't know what it performs like or what its limitations are, but reducing a set of files like the following: 1.jpg 1.dat 1.png 1.txt to: 1.jpg 1.jpg:effect 1.jpg:thumbnail 1.txt:description means all the data related to 1.jpg could now be held in one file. This means moving, copying, deleting files could be done on a single file basis. If I later needed to add a PDF file to this list then I would envision that adding "1.jpg:pdf" would cause less code changes than "1.pdf" as move, copy, delete operations already act upon the parent. I would also surmise that code required to ensure data integrity (eg. if 1.jpg is renamed to 2.jpg then all associated files must be renamed to 2.???) becomes simpler. These possibilities, and the ability to have db-file.csv:key, allow me to look at simpler ways of arranging and managing my data. If it works [well], great; if not, at least it was a learning experience. If others shrug at all this and think "big deal", or, "been there, done that" then that's fine by me . . . the beauty of REBOL is that different aspects of it [and indeed the problem domain] will appeal to different folks at different times. ;) Enjoy! Regards, Ashley

 [14/23] from: atruter:hih:au at: 7-Aug-2002 17:26


Sunanda,
> But it also makes for an application that can't move from the platforms
that
> support streams.
Yeah, I should have said, "... in terms of what I now know NTFS is capable of, and the fact that REBOL transparently handles this [on WindowsNT, 2000 and XP]". Encapsulating stream usage like you suggest, and taking advantage of it on platforms that support it, sounds like a good approach to me IMHO. Even in a Windows only shop you have to deal with the fact that while servers probably run with NTFS there will be at least *some* Win95/98/ME clients out there, or devices that are FAT16/FAT32 based. Regards, Ashley

 [15/23] from: robert:muench:robertmuench at: 7-Aug-2002 8:56


> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]
<<quoted lines omitted: 8>>
> the filesystem, then I can imagine accessing their > contents directly using path notation:
IMO archives solve an other problem: storage space. While streams solve storage access. An archive always has to be handled as one piece. You can't just add a file without "re-compressing" the whole data-stream if you want to keep good compression levels. Streams let you look at one file from the outer world and on to many data-tracks from your application. It's more like a video-tape. You see pictures and hear sound at the same time and you don't care about it. You move the whole tape. Did you ever just listen to the sound track or only viewed the movie track? I didn't.
> File streaming was obviously designed to solve some > specific problems, so I am not criticising it > as such, just the excited reaction as if it is > some wonderful new way of expressing relationships > between clumps of data.
What I find exciting is that we can use this feature from Rebol and that it's very simple to use. So what's the deal? In return you get some decreasing in "data clump" handling with different files. It just makes live easier.
> File streams just add some special leaves to the > branches of the hierarchical filesystem tree.
Yes, that's true but it's a level only the application controls. Up to file level the user can do what he wants, on stream level the application is doing what it wants. As said by others, it can help to keep data integrety consistent. Robert

 [16/23] from: gschwarz:netconnect:au at: 7-Aug-2002 23:30


Robert, The Book is called, "Show-Stopper!" by G.Pascal Zachary 1994 Sub title "The breakneck race to create Windows NT and the next generation at Microsoft" It is a good read. Regards, Greg ----- Original Message ----- From: "Robert M. Muench" <[robert--muench--robertmuench--de]> To: <[rebol-list--rebol--com]> Sent: Tuesday, August 06, 2002 7:05 PM Subject: [REBOL] Re: NTFS Stream Fun I don't remember the title of a book (if you know it let me know it too) that tells the story about the NT

 [17/23] from: robert:muench:robertmuench at: 7-Aug-2002 17:25


> -----Original Message----- > From: [rebol-bounce--rebol--com] [mailto:[rebol-bounce--rebol--com]]
<<quoted lines omitted: 5>>
> Sub title "The breakneck race to create Windows NT and the > next generation at Microsoft"
Hi Greg, thanks a lot :-)))))! I really couldn't remember the title, just to few show-stoppers in my progams ;-). Robert

 [18/23] from: joel:neely:fedex at: 7-Aug-2002 14:14


Hello, all, I've been following this whole discussion with an acute sense of irony that provokes the following comments. No offense intended to anyone... There was a discussion on the list a few months back about the notion of the source code for REBOL eventually becoming available. Someone asserted the view that this would be A Bad Thing since it would tempt/allow people to begin creating incompatible versions of REBOL, thus destroying the platform-independence that is such a desirable feature of the language. My response was that REBOL already offers the means to write platform-specific scripts that cannot be ported. The thing that keeps our scripts transparent/free/portable/virtuous is our own willingness/discipline to avoid using ANYTHING that locks us into a proprietary environment. The best way to avoid becoming Nazgul is to decline Sauron's offer of a ring, no matter how shiny (and powerful) it may be. -jn-

 [19/23] from: al:bri:xtra at: 8-Aug-2002 16:50


> The best way to avoid becoming Nazgul is to decline Sauron's offer of a
ring, no matter how shiny (and powerful) it may be. Embrace, extend, nazgul-ize... Andrew Martin Why yes, my Rebol script works best with MSIE 6... :) ICQ: 26227169 http://valley.150m.com/

 [20/23] from: brian:hawley at: 9-Aug-2002 17:03


At 11:05 AM 8/6/02 +0200, Robert Muench wrote:
>However, IIRC you can have a lot of streams in a NTFS file. BTW: The >stream feature is AFAIK available since NT 3.1 (1994). What you see in a >directory listing is the so called "default" stream. This stream has the >form of <filename>:$DATA
Isn't that supposed to be <filename>::$DATA ? The three fields are: <filename>:<stream>:<attribute> The default stream name is "" (without the quotes). The default attribute is $DATA. All attributes start with $ . Oh wait, I just remembered that since all attributes start with $, you can skip the stream name if you are just using the default and want to specify the attribute. So I guess that <filename>:$DATA would work after all :)
> > In other words, <BFS> completelly smokes NTFS out. :) >NTFS is not that bad. I like some features of BFS as well but IMO the
<<quoted lines omitted: 3>>
>the people and up & downs while developing NT. There is a section about >the development efforts that went into NTFS etc. Robert
The database search capabilities of BFS are where it smokes NTFS. But not for long: Future versions on Windows are slated to have the filesystem built on top of the SQL Server engine, rather than the other way around. When that happens, the smoke will definitely be blowing in the other direction! Brian

 [21/23] from: bga:bug-br at: 9-Aug-2002 22:34


> > > In other words, <BFS> completelly smokes NTFS out. :) > >
<<quoted lines omitted: 13>>
> the other way around. When that happens, the smoke will definitely > be blowing in the other direction!
Actually, no. First of all, the first version of BFS was a real database and it was dropped exactly because the performance hit on using a real database as the filesystem is too big (sure system ara faster today than they were in 1990, but this is no reason to make then slower). Second, BeOS has node monitoring capabilities so you always know rigth on the spot if a file that you need (and you're monitoring it) has changed and you can take appropriate action, This all without having to pool for the file (you receive a notification when it changes). Besides, BFS implements the live query mechanism where you are notified when files stop satisfying your query or when new files start to satisfy them. For instance I have a folder called "Best Musics" that contains all my MP3 files I gave a rating of 10. This folder is *ALWAYS * up to date even if you add and remove files. I am not saying NTFS is not good for what it does, but I am saying BFS is the best "user-oriented" fs *EVER*. Even SGI's XFS does not match it feature-wise and XFS rocks. If you have interest in OS development and FSs is general, take a look at http://www.bug-br.org.br/openbfs. I am the team lead for the OpenBFS project (that aims to create an open-source version of BFS) that is part of the bigger OpenBeOS project (http://www.openbeos.org). Look at the code and you will see why BFS rocks. :) BTW, OpenBFS is, right now, *BETTER* than BFS in some aspects. -Bruno -- Fortune Cookie Says: I'd love to go out with you, but I have to floss my cat.

 [22/23] from: brian:hawley at: 9-Aug-2002 22:02


At 10:34 PM 8/9/02 -0500, Bruno Albuquerque wrote:
> > The database search capabilities of BFS are where it smokes NTFS. > > But not for long: Future versions on Windows are slated to have
<<quoted lines omitted: 15>>
>contains all my MP3 files I gave a rating of 10. This folder is *ALWAYS >* up to date even if you add and remove files.
Strangely enough, all that was exactly what I was referring to when I agreed that BFS smoked NTFS. I just simplified so that I wouldn't get too far off-topic. Thanks for the links, though. I hadn't read _all_ of them ;) As for the future, we'll have to wait for it. I can't remember where I put that set of Yukon and Whistler links, so I can't go into much detail, but there is much to look forward to. We'll have to see what Microsoft can do with the best coders that obscene amounts of money and evil can buy. Maybe they'll surprise us and come up with something anywhere close to what their research papers claim. Brian Hawley

 [23/23] from: greggirwin:mindspring at: 10-Aug-2002 12:11


<< Future versions on Windows are slated to have the filesystem built on top of the SQL Server engine, rather than the other way around. >> The more things change, the more you see how far ahead of their time are/were people like Dick Pick. --Gregg

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted