ILDA Format

[Pangolin]

Format 3 functions as a palette of sorts and it relies on, and enhances, the use of formats 0 & 1.

If I understand everything that you have written, then you admit that Format 3 really isn't a palette at all in the clasic sense of the word, or in the same way that Format 2 is a palette.

Basically, it can be said that "palettes can live on their own". Format 2 in and of itself conveys useful information. Format 3 can't live without some other packet somewhere. Format 3 can not stand on its own and convey useful information.

This is an interesting observation! Format 0, 1, 2, 4 and 5 can all "stand on their own" and convey useful information. Format 3 is the odd man out, in that there must be another packet somewhere -- namely Format 0 or Format 1 to give the data within Format 3 some meaning.

It hasn't been discussed heavily (or possibly at all) up until now, but some of the undesirable artifacts of Format 3 were indirectly discussed in your description above. Format 3 opens up the possibility for bugs. If you find a Format 3 packet, you MUST see either a Format 0 or Format 1 packet as the next thing you find -- otherwise, the file must be declared as "having a bug". Also, you have to look at the number of colors found in the Format 3 packet, and see if that corresponds to the number of points in the Format 0/1 packet and if not, then the file must be declared " having a bug". This "possibility for bugs" was discussed within the email thread in the ILDA Tech Com list years ago, and also in the various Technical Committee meetings. Basically, these were seen as drawbacks, especially since I don't think there is ANYONE among us who hasn't seen an incorrectly-implemented ILDA file, and so surely there would be bugs with people implementing Format 3... With the other problems of Format 3 (larger file size and inability to stream), these are some of the reasons why Format 3 was eventually abandoned.

Format 2 and format 4 are totally incompatible and should never be found together in any logical formation of an ILDA file. Format 2 is JUST a palette and format 4 is a set of points with colors.

Now there’s a word you really have to watch: "never". Below the words "best" and "always" appear. In my experience, you really have to watch these words...

Nevertheless, Format 4 and 2 could conceivably be used together in the way that you are trying to use Format 3/0 and 2. Would this be the most logical use of Format 4 and 2 together? Probably not. But, if I understand how you are using Format 3 data, it appears to me as though the Format 4/2 could be used in exactly the same way.

In reading your explanation, I can see that you are using Format 0/3 data in a way it was never really meant to be used. If I understand you correctly, you make a "best fit palette" against Format 3 data, trying to create indexes out of the frame from Format 0 and possibly a Format 2 palette. But this could be very foolish. For example, if you have frame data with Format 3/Format 0 data, the Format 0 portion is not guaranteed to have any useful data at all for the "color index" portion.

As POSSIBLE proof of this (I don't have any idea if it is proof at all because I never investigated this but), remember those test frames that Matt and Peter Jakubek sent you that made you so upset (the ones where they said “here are the example frames for Format 3”)? Well, what data was in the "color index" portion of the Format 0 section? Was it useful data that could have been used to best-fit the actual colors from Format 3? Or was it simple "the beam is on here" and "the beam is off over there" type data? With Format 3, you are only guaranteed the latter...

Basically, it appears to me that you are finding some sort of utility from using real palettes with what are supposed to be true-color frames and combining them in some way. Nobody in ILDA ever had this intention. This is all your idea. It kinda shows what a unique thinker you are, and I kinda like that .

But I would liken this to using a butter knife as a screwdriver. I do it occasionally, as needed, but a butter knife was never intended to be used as a screwdriver. You won't find advertisements telling you how great a company's butter knife works as a screw driver... That's not the intention, and that's not what it's really good at. There are better tools to accomplish thd job.

Photos look the best as jpg files. Logos look best as gif or png.

There's one of those words you really have to watch -- "best". It conveys absolution and objectivity, but too often is used subjectively...

For example, I could say that LD2000 is the "best" for creating laser shows, but the problem is, "best" in what way? If absolute cheapest price is your priority, then it isn't the "best".

Likewise, I would say Photos don't LOOK BEST as JPG files! Photos LOOK BEST with some kind of high-dynamic-range lossless format. BUT these files are damned large in size! Photos might WORK BEST for the average consumer if they are JPEG, but this has nothing to do with the "look". It’s more a question of priority.

And at Pangolin, we use the 24-bit PNG format all of the time to convey continuous tone, photographic images as well as logos, because PNG still offers good compression, but is lossless. For Pangolin, 24-bit PNG is the "best", even for most of the photos we use around the office. But since the file size is larger than JPG, if file size is your priority, then it isn't the "best".

As you might already know... jpg files are considered to be "true color" and gif and png are always "palette".

There's another word you have to watch out for -- "always"...

There is a 24-bit PNG format, and we use it all the time at Pangolin. Please see the following from Wikipedia:
http://en.wikipedia.org/wiki/Portable_Network_Graphics

PNG files CAN! use a palette, but don't necessarily do so all of the time. Within PNG files, palettes are not expected to be used for "true color" and "true color with alpha" images.

There is absolutely NOTHING outdated about the use of color palettes!

For data storage means, the use of Palettes CAN (not necessarily "do", but "can") reduced the file size of certain files. For example, if you have an ILDA file with 10 frames, the first chunk being Format 2, and the rest being Format 0, this would be smaller than if all 10 frames were Format 4. BUT the file size would be larger if all 10 frames were encoded as Format 3/0 types. That's one of the down-sides of Format 3.

Nevertheless, I really do think we are, at least close to the end of this discussion. We have discovered that you use Format 3 and color data in a way it really wasn’t intended to be used, and in a way that it has never been documented (and nobody else uses). This is pretty unique to LaserBoy, and it may allow for special effects that no other software allows. For this, I applaud you!

But I don't see this novel use as justification for retaining Format 3, especially since you could accomplish what it is that you are doing in other ways (several other ways in fact). Also, if you had your wish to come true, ILDA would view it as "a new Format 3" since Format 3 with the old header was never published as a preliminary standard or any kind of standard.

I guess in the end, where we are at with this is that LaserBoy has some unique features and James should get some kudos for that! But I think the best approach to exploiting those features must surely be in using something other than Format 3. Otherwise, actually it is no problem for you to keep using Format 3 since it is already coded – just use the “il3” file extension or something like that.

If other companies see the same benefit you do with such a use, then they would surely follow. After all, if LaserBoy really could so some wazmo things that were highly desirable and no other software on earth could do, this would create a kind of “vacuum” in the market, and the free market shows us that this “vacuum” would have to be filled, and sooner or later, other platforms would implement (or copy) your technique. God knows that’s happened enough with our software...

Bill

[James Lehman]

Your observations about the possibility of a file "having a bug" don't really make any sense. ANY file with erroneous data in it has a bug!

There are many examples of files that have the quantity of elements stored in the header, followed by THAT number of those elements. If those numbers don't match, that's a bug.

That issue is dealt with in the software that reads the file. What do you do when these things don't match? This is NEVER a reason to limit a storage structure. Otherwise, there would be no computer science!

In my reality, no, a format 2 does not make any sense with no format 0 or 1 after it. LaserBoy can read them from a file that way. But there is no way to use LaserBoy to save it like that.

You can not store an 8 bit bitmap with the palette and no image data!

YES format 3 IS a palette, of sorts. The only difference between it and format 2 is that the maximum number of RGB values can be more than 256.

Format 0 is an ordered set of vertices that each contain {X, Y, Z, K, C}
X, Y, Z are signed short integer (-32768 to 32767)
K, C are unsigned byte (0 to 255)
K contains a bit that indicates blank or not blank.
C is the one byte index into a format 2 palette.

Format 1 is an ordered set of vertices that each contain {X, Y, K, C}
X, Y are signed short integer (-32768 to 32767)
K, C are unsigned byte (0 to 255)
K contains a bit that indicates blank or not blank.
C is the one byte index into a format 2 palette.

Format 2 is a list of byte indexed {R, G, B}.
R, G, B are unsigned byte (0 to 255)
There can be no more than 256 elements

Format 3 is a list of natural order {R, G, B}.
R, G, B are unsigned byte (0 to 255)
There can be no more than 65536 elements

Format 4 is an ordered set of vertices that each contain {X, Y, Z, K, B, G, R}
X, Y, Z are signed short integer (-32768 to 32767)
K is unsigned byte (0 to 255)
K contains a bit that indicates blank or not blank.
B, G, R are unsigned byte (0 to 255)
NOTE: THERE IS NO C !!!

Format 5 is an ordered set of vertices that each contain {X, Y, K, B, G, R}
X, Y are signed short integer (-32768 to 32767)
K is unsigned byte (0 to 255)
K contains a bit that indicates blank or not blank.
B, G, R are unsigned byte (0 to 255)
NOTE: THERE IS NO C !!!

Format 2 + (0 or 1) is a complete set of <=256 colors and image.

Format 3 + (0 or 1) is a complete set of >256 colors and image.

Format 2 + 3 + (0 or 1) is a complete set of >256 colors and image ~ with an automatic, retro-engineered fallback with palette indices into the preceding format 2.

Format 2 + 4 is an orphan palette and an image with its own colors and NO PALETTE INDICES!

There are MANY ways to do color reduction to get a best match between a set of RGB values that number more than 256 to a reasonable set of RGB values that number less than or equal to 256.

Just one way to do it:

Take only the first 3 bits of red, the first 3 bits of green and the first 2 bits of blue. 3 + 3 + 2 = 8 bits.

Again, I would ask you to look at this:

http://www.akrobiz.com/laserboy/ilda_file_format.html

Scroll to the bottom and read the last few paragraphs. Then look at the example pictures.

EZFB_332 is a packed pixel palette.

You can hardly see the difference between that and true color.

With a more subtle version of color reduction, you would not be able to see the difference at all.

James.

[keeperx]

well if we are keeping it technical, then im out..

i have a question for Bill, and im only asking because i actually don't know.

What are the benefits of the format 4/5 color storage methods over the format 3 method?

I mean actual benefits. and again im not being a smart ass, i just really do not know enough about it.

is it:
faster (whatever that means)
smaller
easier to read/write
stuff like that...

I never really got an answer to this question.



also if format 4/5 stores information in a "1 to 1" relationship between points and color, and a format 0 or 1 plus format 3 stores that data in a "many to 1" relationship between points and color, i would think that a the latter would have a smaller file size. Furthermore it would make it easy to change a single color globaly thru the entire file.

again im probably wrong, im batting 1000 here.

[James Lehman]

Format 2 has a one-to-many relationship with formats 0 or 1.

Format 3 has a one-to-one relationship with formats 0 or 1.

The difference in size between a format 3 + 0 vs. a format 4 is this:

Format 3 = 32 byte header + 3 byte {R, G, B}
Format 0 = 32 byte header + 8 byte {X, Y, Z, K, C}

Format 4 = 32 byte header + 10 byte {X, Y, Z, K, B, G, R}

So for every frame with true-color you save in formats (3 + 0) you need to save two 32 byte headers and 11 bytes per point.

In format 4 you need only save one 32 byte header and 10 bytes per point.

That's a difference of 1 byte per point per frame!

Big deal!

That one byte is the palette index that makes 0, 1, 2 & 3 a perfect set!

Zip it and you won't see the difference!

When you use formats 0, 1, 2 & 3 you get all of the benefit of being able to store BOTH true color and palette data in the same file!

When you use formats 4 or 5, you lose that benefit. You also lose ALL compliance and conformity to formats 0, 1, 2 & 3.

James.

[James Lehman]

Bill, would you please describe your understanding of the use of format 2?

James.

[Pangolin]

I never really got an answer to this question.

I have already written about them a few times, but let me see if I can make a buletted list:
* Conforms to the original intention of the format bit pattern proposed many years ago, including a bit for "dynamic focus" (which, up until now has not been discussed)
* Far less potential for bugs since the color data is integrated with the X-Y-Z data
* Smaller file size when compared to Format 3/0 or 3/1
* Data "alignment" (i.e. even number of bytes per entity), which could be an issue for certain microprocessors
* Ability to stream data directly from the file (could be beneficial for "memory players")

These are benefits off the top of my head. There could be others that I am not thinking of right now. All of the benefits and drawbacks of the various formats were discussed in the public emails over the ILDA Tech Com list server, which was generally public information available and viewable by anyone.

Interestingly enough, the only drawback of Format 4 is discussed by James below -- but it's only a drawback if you view Format 3 in a way it was never conceived to be viewed.

When you use formats 0, 1, 2 & 3 you get all of the benefit of being able to store BOTH true color and palette data in the same file!

But it was never intended to work that way in the first place. This is your own incarnation. It is pretty clever if you are lucky enough to have Format 0/1 data that has valid palette indexes, but nobody within ILDA ever conceived of such a thing. Basically, when you use Format 3, the "color index" in 0 or 1 wasn't even guaranteed to be valid!

When you use formats 4 or 5, you lose that benefit. You also lose ALL compliance and conformity to formats 0, 1, 2 & 3.

Well I guess "compliance" and "conformity" are in the eye of the beholder. If Format 4/5 weren't "compliant" and didn't "conform" to formats 0 and 1, then I wouldn't be able to describe to someone how to implement Format 4 and 5 with only two sentences (which I have routinely done).

Also, your statement does not reconcile with the original intention of the ILDA format bit pattern. (Remember, the format code is bit-encoded, not index-encoded.) Format 3 is a bastardization of that pattern, proposed by someone who was ignorant of how the format code was supposed to work. Really it is Format 3 that is not compliant with the original ideas of how the format would evolve!

I had written a lot more about this topic, and the origin of Format 3 but... something happened to the computer and it was lost. And I don't feel like re-typing it. (I think we have all been there at one time or another with forum software...)

Suffice it to say that I am really out of gas on this topic, and I think the best thing to do is to just see how things shake-out in the marketplace.

At the next SELEM or other PL gathering, it would be good to have a more complete demonstration of LaserBoy and what can be done with color. I think you would agree with me on that one .

Bill, would you please describe your understanding of the use of format 2?

I think you and I are in agreement about what Format 2 is and what it does.

Briefely, Format 2 really is a palette in the true sense of a word. A file can have only Format 2 data if it wants to. That is a legal "combination" and the ILDA file spec says as much. Format 2 can precede some number of frames, and that number can be as few as "1" (or zero as stated above). Not much more to it than that...

Format 2 was an attempt to correct two past mistakes in the ILDA format conception -- both mistakes being color related. But it wasn't really strong enough, and, despite Format 2 being around for a relatively long time, almost nobody implemented it. That's why Format 4/5 was invented, to be a true and total solution to the problem of communicating X-Y-Z and color information. You might say that with Format 4 you "lose data", but the only data that is lost is the palette index, which is the entire goal in the first place -- to get rid of palettes.


Bill

[James Lehman]

Think of it like this... We all know that formats 0 & 1 have been out there for a long time. If we know we are writing new files with format 3, we can make sure they have whatever palette index information in them we want in the format 0 or 1. If we encounter a format 3, while reading a file that comes from somewhere else, we know what to do with it and we know we don't care if it has proper indices into any format 2 palette! That's just a possible retro fit to older ILDA readers that we know can't read format 3.

This is definitely forward thinking that does not wreck the past!

If software that reads or writes the current ILDA file standard has bugs in it, FIX IT!

Fearing that you might be wrong is not a strong argument that you are right. Just because a few pieces of data are not stored right next to each other in a file is not an excuse to throw away the whole idea.

I am an artist first and an engineer second. I have no problem saying that.

I learned a long time ago that it makes no sense to fight with and try to force any medium of art to be something that you want it to be.

The only way to make truly outstanding art is to let the medium be what it is and let it teach you how it works.

A good example is polymer clay! It is a fairly new and untested medium of art. It has its own characteristics, like no other clay or paint. If you force it into making impossible things it will always fail! If you work with it and learn how it works with you, then you can help it become something beautiful.

http://akrobiz.com/pc/

The ILDA file format is nothing more than a man made arrangement of numbers in linear order; a clay of sorts.

WHO CARES WHAT THE ORIGINAL INTENTION WAS?

I can NEVER know that!

I can, however, know the ways of binary data and computer science.

As with any other medium of art, the masters are the ones who figure out how to get every last nuance, trick, exploit and benefit out of their medium!

If no one else ever saw the possible benefits of doing it this way, so be it. I'll take credit for that! But there it is. I just pointed it out. So let's go with it now. I would be thrilled to write the next ILDA file standard document. Seriously.

I don't know how much more "compliant or conforming" you can get when you compare format 0 to 0 and 1 to 1! It's really that simple. Format 0 is not the same as 4 and format 1 is not the same as 5! If you are looking for only formats 0 & 1 you will NEVER find them in a file full of 4 & 5!

I am really glad we have been able to get into the details of this subject on a well read forum! I hope you really are not "out of gas on this topic". I hope you and others don't look at it as an unfortunate done-deal. It is not yet carved into stone! ILDA has not yet put the red wax seal and the fancy ribbons on this as the final constitution of laser art.

Getting rid of palettes is a BAD IDEA! Let's get together on this. I have already proved it!

I can read a lot between the lines and I know that you (Bill) and I do laser signals very differently. My computer is my CPU, system RAM and hard drive. Your computer is a special card that has its own CPU and RAM. I need to figure out lots of stuff and then shoot it from a laser. You need to store a few static states and figure stuff out on-the-fly on your own computer card. We see things from a very different starting point.

I really do have A LOT to add to human knowledge about the art of laser display.

Let's stop butting heads.

James.

[James Lehman]

I just noticed that this thread has picked up a new tag... "still egotistical...".

Am I really being egotistical when I'm (still) just trying to get a point accross that I've been working on and aware of for about 4 years?

BTW I offered all of this information to ILDA back when I first ran into it. I also offered to help with the standard then too.

James.

[platinum]

I just noticed that this thread has picked up a new tag... "still egotistical...".
. . .

Points for slyness to whoever did that, at least.

I've read through 98% of this thread (I admit I skimmed a couple of paragraphs overall) - and I think it's been pretty tame overall.

I must also point out that thanks to you and Bill going into so much detail on all of this, I've learned a lot about ILDA formats and some things about Laserboy, especially in regards to how it uses ILDA files.

-Jonathan

[Zoof]

That should cover the color part of the ILDA standard. Now lets move onto the next topic:
"Why the IDLA format is a really poor way of storing vector art"